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PREFACE 


The  Air  Force  is  in  the  process  of  implementing  performance-based 
practices  within  its  service  contracts  to  improve  service  quality  and  reduce 
costs.  RAND's  Project  AIR  FORCE  is  supporting  these  efforts.  Our  early 
research  focused  on  installation  support  services.  Recently,  the  Deputy 
Assistant  Secretary  for  Contracting  (SAF/ AQC)  asked  us  to  include 
purchased  services  that  support  weapon  system  development  and 
sustainment  activities  as  well.  This  Documented  Briefing  describes  an 
analysis  of  the  application  of  performance-based  practices  in  service 
contracts  at  an  Air  Force  Air  Logistics  Center  and  a  Product  Center.  It  is 
part  of  the  Project  AIR  FORCE  study  "Improved  Implementation  of 
Performance-Based  Services  Acquisition:  Managing  Performance  and 
Assessing  the  Effects  of  Practices,"  sponsored  by  SAF/ AQC.  It  should  be 
of  interest  to  the  Air  Force  and  Department  of  Defense  communities. 

For  almost  a  decade,  RAND  has  been  helping  the  Department  of  Defense 
improve  the  way  it  purchases  goods  and  services.  Readers  may  also  be 
interested  in  selected  related  studies: 

•  Implementing  Best  Purchasing  and  Supply  Management  Practices: 
Lessons  from  Innovative  Commercial  Firms  by  Nancy  Y.  Moore,  Laura 
H.  Baldwin,  Frank  Camm,  and  Cynthia  R.  Cook,  RAND  DB-334- 
AF,  2002,  which  can  be  downloaded  from  www.rand.org/ 
publications  /DB  /DB334 

•  Federal  Contract  Bundling:  A  Framework  for  Making  and  Justifying 
Decisions  for  Purchased  Services  by  Laura  H.  Baldwin,  Frank  Camm, 
and  Nancy  Y.  Moore,  RAND  MR-1224- AF,  2001,  which  can  be 
downloaded  from  www.rand.org/publications/MR/MR1224 

•  Performance-Based  Contracting  in  the  Air  Force:  A  Report  on 
Experiences  in  the  Field  by  John  Ausink,  Frank  Camm,  and  Charles 
Cannon,  RAND  DB-342-AF,  2001,  which  can  be  downloaded  from 
www.rand.org/publications/DB/DB342 

•  Strategic  Sourcing:  Measuring  and  Managing  Performance  by  Laura  H. 
Baldwin,  Frank  Camm,  and  Nancy  Y.  Moore,  RAND  DB-287-AF, 
2000,  which  can  be  downloaded  from  www.rand.org/ 
publications  /DB  /DB287 


•  Incentives  to  Undertake  Sourcing  Studies  in  the  Air  Force  by  Laura  H. 
Baldwin,  Frank  Camm,  Edward  G.  Keating,  and  Ellen  M.  Pint, 
RAND  DB-240-AF,  1998 

•  Strategic  Sourcing:  Theory  and  Evidence  from  Economics  and  Business 
Management  by  Ellen  M.  Pint  and  Laura  H.  Baldwin,  RAND  MR- 
865- AF,  1997. 

Research  on  services  acquisition  and  broader  purchasing  and  supply 
management  policy  in  the  Air  Force  continues  within  the  Resource 
Management  Program  of  Project  AIR  FORCE.  For  additional  information 
or  to  convey  comments  on  this  document,  please  contact  Dr.  Laura  H. 
Baldwin  at  (412)  683-2300  x4901  or  at  Laura_Baldwin@rand.org. 

For  information  about  RAND,  please  see  our  website,  www.rand.org. 


Project  AIR  FORCE 

Project  AIR  FORCE,  a  division  of  RAND,  is  the  Air  Force  federally  funded 
research  and  development  center  (FFRDC)  for  studies  and  analyses.  It 
provides  the  Air  Force  with  independent  analyses  of  policy  alternatives 
affecting  the  development,  employment,  combat  readiness,  and  support  of 
current  and  future  aerospace  forces.  Research  is  performed  in  four 
programs:  Aerospace  Force  Development;  Manpower,  Personnel,  and 
Training;  Resource  Management;  and  Strategy  and  Doctrine. 
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SUMMARY 


In  April  2000,  Dr.  Jack  Gansler,  Under  Secretary  of  Defense  for  Acquisition 
and  Technology,  established  the  goal  that  at  least  50  percent  of  all  service 
acquisitions,  measured  in  dollars  and  contracts,  should  be  performance- 
based  by  2005.  Air  Force  interest  in  performance-based  service  contracts 
preceded  Dr.  Gansler's  memorandum.  On  April  1, 1999,  the  Air  Force 
issued  Air  Force  Instruction  (AFI)  63-124,  Performance-Based  Service 
Contracts  (PBSC),  which  contains  guidance  on  implementing  performance- 
based  practices  for  purchasing  a  wide  range  of  services  to  support  its 
installations,  employees,  and  war-fighting  capability.  Under  what  is  now 
called  performance-based  services  acquisition  (PBS A),  buyers  should  (1) 
describe  what  service  is  desired  and  not  how  to  do  it,  (2)  use  measurable 
performance  standards  and  quality  assurance  plans,  (3)  specify 
procedures  for  reductions  in  fee  or  price  when  services  do  not  meet 
contract  requirements,  and  (4)  include  performance  incentives  where 
appropriate. 

Previous  RAND  research  has  supported  the  implementation  of  PBSA 
practices  in  the  acquisition  of  installation  support  services  purchased 
through  operational  contracting  activities.  In  March  2001,  SAF/AQC 
asked  RAND  to  expand  its  research  scope  to  support  ongoing  Air  Force 
efforts  to  implement  PBSA  for  services  purchased  by  the  Air  Force 
Materiel  Command  (AFMC)  that  are  related  to  the  acquisition  and 
sustainment  of  weapon  systems.  This  Documented  Briefing  presents  what 
we  learned  about  the  application  of  performance-based  practices  in  these 
service  areas  during  interviews  at  an  Air  Logistics  Center  (ALC)  and  a 
Product  Center.1 

An  important  lesson  from  the  interviews  is  that  the  nature  of  the  services 
purchased  within  the  "systems"  sides  of  these  two  Centers — that  is, 
program  offices  that  support  weapon  systems,  common  subsystems,  and 
special  mission  capabilities — differs  from  that  of  installation  support 
services  in  ways  that  affect  the  implementation  of  AFI  63-124.  Many 
installation  support  services  (e.g.,  grounds  maintenance)  are  commercial 
services  with  accepted  performance  standards  and  robust  commercial 


Assurances  of  anonymity  for  the  personnel  we  interviewed  prevent  us  from  further 
identification  of  these  Centers. 


markets.  Performance  of  such  services  can  often  be  measured  objectively 
and  repeatedly  over  time.  In  contrast,  many  of  the  services  purchased  on 
the  systems  sides  of  these  Centers  allow  only  infrequent  opportunities  for 
performance  evaluation,  deal  with  tasks  for  which  it  is  difficult  to  define  a 
"successful"  outcome  ex  ante,  or  lend  themselves  only  to  subjective 
evaluations.  Additionally,  proprietary  data,  rapid  evolution  of 
technology,  and  limited  demand  for  some  of  these  services  limit  the 
competition  for  their  provision.  As  a  result  of  these  distinctions  and  the 
fact  that  the  examples  in  the  current  version  of  the  AFI  focus  on 
installation  support  services,  personnel  perceive  some  ambiguity  about 
the  definition  of  a  "service"  for  purposes  of  the  AFI. 

Many  at  the  ALC  and  the  Product  Center  feel  that  it  is  difficult  or 
impossible  for  systems  contracts  to  satisfy  all  four  of  the  PBSA  criteria 
described  in  AFI  63-124.  In  particular,  they  find  it  difficult  to  satisfy  the 
requirement  to  use  "measurable  performance  standards."  They  interpret 
this  to  mean  that  the  desired  result  of  a  service  must  be  known  in  advance 
and  that  objective  data  must  be  collected  frequently  to  measure 
performance  against  that  result.  This  interpretation  cannot  be  applied 
easily  to  many  services  purchased  on  the  systems  sides  of  these  Centers. 
Despite  this  difficulty,  however,  both  Centers  use  a  performance-based 
approach  (applying  the  other  three  criteria)  to  purchase  many  services, 
and  many  personnel  feel  that  they  can  determine  and  convey  whether  a 
contractor  met  their  needs.  As  a  result,  we  conclude  that  many  of  the 
approaches  used  by  the  ALC  and  the  Product  Center  satisfy  the  intent  of 
AFI  63-124. 

Because  of  the  nature  of  services  purchased  by  these  Centers  and  some 
confusion  about  the  interpretation  of  PBSA  practices,  the  Air  Force  could 
improve  implementation  of  AFI  63-124  by  clarifying  the  universe  of 
services  to  which  it  applies.  The  Air  Force  could  also  provide  examples  of 
what  kinds  of  performance  objectives  and  monitoring  activities  satisfy  the 
criterion  for  "measurable  performance  standards"  (or  serve  as  substitutes) 
in  the  context  of  services  for  which  objective  measures  of  success  are 
difficult  to  develop. 
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1.  INTRODUCTION 


Implementing  Performance-Based 
Services  Acquisition  (PBSA): 

Perspectives  from  an  Air  Logistics 
Center  and  a  Product  Center 

John  Ausink,  Laura  H.  Baldwin, 
Sarah  Hunter,  Chad  Shirley 


During  April  and  May  2001,  we  visited  an  Air  Force  Air  Logistics  Center 
(ALC)  and  a  Product  Center  to  explore  the  use  of  performance-based 
practices  in  their  service  acquisition  activities.2  We  decided  to  conceal  the 
identities  of  these  two  Centers  to  encourage  personnel  to  share  the 
challenges  as  well  as  the  successes  they  are  encountering  in  implementing 
new  practices.  This  Documented  Briefing  describes  what  we  learned 
during  our  interviews  and  through  review  of  solicitations,  contracts, 
surveillance  documents,  and  incentive  plans  associated  with  selected 
purchased  services. 


2The  Air  Force  has  three  ALCs  (Ogden  ALC,  Oklahoma  City  ALC,  and  Warner  Robins 
ALC)  and  four  Product  Centers  (Air  Armament  Center,  Aeronautical  Systems  Center, 
Electronic  Systems  Center,  and  Space  &  Missile  Systems  Center). 


1 


To  the  extent  that  experiences  at  these  two  Centers  are  representative  of 
the  Air  Force's  other  Centers,  the  lessons  that  we  draw  from  these 
interviews  should  be  broadly  applicable.  We  believe  this  is  the  case  and 
assume  that  the  other  Centers  would  provide  additional  insights. 


2 


We  Learned  About  Purchased 
Services  and  PBSA  Opportunities 


•  Early  Air  Force  PBSA  implementation 
efforts  focused  on  installation  support 
services 

•  AFMC  buys  many  services  that  support 
weapons  and  other  mission  capabilities 

•  SAF/AQC  wants  to  ensure  that 
performance-based  practices  are 
implemented  Air  Force-wide 

RAND  Project  AIR  FORCE 


In  April  2000,  Dr.  Jack  Gansler,  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology,  established  that  a  minimum  of  50  percent  of 
Department  of  Defense  (DoD)  service  acquisitions,  in  both  dollars  and 
contracts,  be  performance-based  by  the  year  2005.  Performance-based 
practices  are  expected  to  help  the  DoD  improve  performance,  innovation, 
and  competition  in  purchased  services,  often  at  a  reduced  cost.3  In 
October  2000,  the  Air  Force  began  tracking  the  use  of  performance-based 
service  contracts  through  a  new  data  field  on  the  DD  Form  350,  Individual 
Contracting  Action  Report,  which  records  information  about  contract 
transactions  over  $25,000. 

Air  Force  efforts  to  implement  performance-based  practices  preceded  Dr. 
Gansler's  memorandum.  In  1999,  SAF/AQC  issued  an  Air  Force 
Instruction,  AFI  63-124,  containing  guidance  for  implementing 


3See  Gansler  (2000). 
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performance-based  services  acquisition  (PBSA)  practices.4  It  is  based  on 
the  Federal  Acquisition  Regulation  (FAR)  Part  37  definition  of  a 
performance-based  service  contract,5  including  (1)  a  description  of  the 
desired  results,  not  the  method  of  provision;  (2)  the  use  of  measurable 
performance  standards  and  quality  assurance  surveillance  plans;  (3) 
provisions  for  reductions  of  price  or  fee  when  a  service  is  not  performed 
or  does  not  meet  contract  requirements;  and  (4)  the  use  of  positive 
incentives,  where  appropriate.  This  instruction  applies  to  virtually  all  Air 
Force  service  contracts  over  $100,000  annually. 

The  Air  Force  purchases  a  broad  range  of  services  to  support  its 
installations,  military  and  civilian  employees,  and  primary  war  fighting 
capabilities.  Initial  Air  Force  PBSA  implementation  efforts  focused  on 
installation  support  services  purchased  through  operational  contracting 
activities.  RAND's  previous  and  current  research  supports  these 
implementation  efforts.6  In  March  2001,  SAF/AQC  asked  us  to  expand 
our  research  scope  to  include  the  services  that  the  Air  Force  Materiel 
Command  (AFMC)  purchases  to  develop  and  support  the  Air  Force's 
weapon  systems.  These  services  were  of  interest  to  SAF/ AQC  primarily 
for  two  reasons.  First,  AFMC  spent  over  $12.5B  on  these  services  in  FY 
2000,  compared  to  just  over  $6.5B  Air  Force-wide  on  operational  services.7 
Second,  AFMC  was  just  beginning  to  implement  AFI  63-124  broadly  due 
to  delays  associated  with  a  union  protest. 


4AFI 63-124,  Performance-Based  Service  Contracts  (PBSC),  April  1, 1999,  is  currently  in 
revision  under  the  new  title  Performance-Based  Services  Acquisition  (PBSA).  The  proposed 
revision  that  the  authors  have  seen  includes  changes  to  clarify  ambiguities  reported  by 
Air  Force  organizations  that  are  implementing  PBSA  practices. 

5See  FAR  Part  37,  Service  Contracting,  Subpart  37.6 — Performance-Based  Contracting. 

6See  Ausink  et  al.  (2001). 

7These  numbers  are  from  an  analysis  of  Air  Force  data  on  FY00  contract  transactions 
over  $25,000.  The  Air  Force  tracks  these  contracts  through  the  DD  Form  350,  Individual 
Contracting  Action  Report.  Service  contracts  were  identified  through  the  Product/ 
Service  Code  (PSC)  associated  with  each  contract.  The  PSC  for  service  contracts  begins 
with  a  letter  rather  than  a  number.  Service  contracts  were  assigned  to  Air  Force 
organizations  based  on  the  office  contracting  code  of  the  purchasing  organization,  which 
is  part  of  the  DD  Form  350  record  for  each  contract.  Services  were  further  classified  as 
related  to  operational  contracting,  sustainment,  or  weapons  development  by  examining 
the  office  contracting  code,  office  name,  and  address.  We  thank  our  RAND  colleagues, 
Nancy  Moore  and  Charles  Lindenblatt,  for  creating  these  classifications  and  performing 
these  data  analyses. 
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Headquarters  AFMC  suggested  that  we  visit  an  ALC  and  a  Product 
Center  to  learn  about  the  kinds  of  services  purchased  and  opportunities  to 
apply  performance-based  practices  in  these  activities. 

In  what  follows,  we  will  provide  examples  of  performance-based  practices 
currently  used  in  services  contracts  at  this  ALC  and  Product  Center  and 
illustrate  how  selected  practices  satisfy  (in  our  opinion)  the  intent  of  AFI 
63-124. 
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We  Interviewed  a  Wide  Range  of  Personnel 
Involved  in  Diverse  Purchasing  Activities 


•  ALC:  Six  Product  Directorates  that 
purchase  services  related  to 
supporting  systems 

•  Product  Center:  Three  Program  Offices 
that  purchase  services  related  to 
acquiring  systems 

•  Operational  contracting  at  each 

•  Contracting,  requirements,  and 
program  management  personnel 

RAND  Project  AIR  FORCE 


We  spent  four  days  at  the  ALC.  During  this  time,  we  met  with  program 
managers  and  other  "requirements  personnel"  (e.g.,  engineers  or  other 
subject  matter  experts)  as  well  as  program  contracting  officers  from  six 
Product  Directorates  that  support  weapon  systems,  common  subsystems, 
and  special  mission  capabilities.  These  and  other  related  Product 
Directorates  form  the  "systems"  side  of  the  ALC.  We  also  met  with 
requirements  personnel  and  contracting  officers  who  participate  in 
operational  contracting  activities  in  support  of  the  installation  and  those 
who  are  responsible  for  implementation  of  AFI  63-124  at  this  ALC. 

We  spent  a  day  and  a  half  at  the  Product  Center.  We  met  with  contracting 
officers  and  program  management  personnel  from  operational  contracting 
and  three  Program  Offices  that  purchase  services  to  support  the  Center's 
system  acquisition  activities.  We  also  met  with  contracting  personnel 
responsible  for  implementing  purchasing  policy  at  this  Product  Center. 

We  discuss  the  types  of  services  addressed  during  these  interviews  in  the 
next  chapter. 
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We  Can  Summarize  Our  Observations 
in  Four  Main  Points 


•  Services  purchased  on  the  “systems”  side  of  AFMC  differ 
from  installation  support  services  in  ways  that  are  important 
for  implementation  of  AFI  63-124 

•  Application  of  AFI  to  certain  services  was  questioned 

•  Both  Centers  already  use  a  performance-based  approach  to 
purchase  many  services 

•  However,  personnel  are  struggling  to  reconcile  their 
surveillance  and  performance  measurement  activities  with 
the  FAR  definition  of  PBSC 

•  In  our  view,  many  of  the  described  approaches  satisfy  the 
intent  of  AFI 

•  SAF/AQC  can  improve  implementation  of  AFI  by  clarifying 
its  application  to  AFMC  systems  services  contracting 

RAND  Project  AIR  FORCE 


First,  services  purchased  by  the  Product  Directorates  and  Program  Offices 
at  these  two  Centers  to  support  acquiring  and  sustaining  weapon  systems 
and  other  mission  capabilities  ("systems"  contracting  activities)  differ 
from  installation  support  services  in  ways  that  affect  the  Air  Force's  efforts 
to  implement  AFI  63-124.  For  example,  many  of  the  systems  services 
involve  rapidly  evolving  or  obsolete  technologies  with  limited  demand, 
resulting  in  limited  competition;  it  is  often  difficult  to  define  in  advance  a 
"successful"  outcome  for  the  service;  there  may  be  infrequent 
opportunities  to  observe  performance;  and  it  can  be  difficult  to  measure 
performance  objectively. 

These  differences,  combined  with  the  fact  that  the  examples  in  the  AFI 
currently  focus  on  operational  contracting  activities,  cause  some  personnel 
at  these  Centers  to  question  whether  the  AFI  really  applies  to  the  kinds  of 
services  they  buy.  In  fact,  we  were  repeatedly  asked  how  we  define 
services  for  the  purposes  of  our  study. 

Although  it  was  not  raised  during  our  interviews,  the  definition  of  the 
universe  of  services  to  which  AFI  63-124  applies  has  broad  implications.  It 
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significantly  affects  how  the  Air  Force  evaluates  its  progress  toward 
meeting  Dr.  Gansler's  50  percent  goal  for  PBSA. 

Second,  many  of  the  services  acquisition  activities  that  were  discussed 
during  our  interviews  reflect  performance-based  practices.  In  particular. 
Product  Directorates  and  Program  Offices  convey  to  their  contractors 
what  they  need,  not  how  to  perform  the  work;  they  can  evaluate  and 
substantiate  whether  the  contractor  is  meeting  those  needs;  and  in  some 
cases,  they  use  positive  and/ or  negative  incentives  to  align  the 
contractor's  activities  with  the  customer's  needs.  Interestingly  though, 
few  attribute  adoption  of  performance-based  practices  to  AFI  63-124; 
rather,  they  credit  acquisition  reform  and  Ms.  Darleen  Druyun's 
"Lightning  Bolts"  for  the  changes  in  practices.8 

Third,  many  of  these  Centers'  current  performance-based  practices  clearly 
match  three  of  the  four  parts  of  the  FAR  Part  37  definition  of  a  PBSA 
activity  used  in  AFI  63-124.  Flowever,  personnel  are  struggling  to  link 
their  surveillance  and  performance  management  activities  to  their  strict 
interpretation  of  the  requirement  for  "measurable  performance  standards" 
to  evaluate  contractor  performance. 

In  spite  of  the  hesitation  of  many  personnel  to  label  their  services  contracts 
containing  performance-based  practices  as  PBSA  activities,  it  is  our 
opinion  that  many  of  the  services  contracting  activities  at  these  Centers 
satisfy  the  intent  of  AFI  63-124. 

Finally,  it  became  clear  to  us  during  our  interviews  that  SAF/AQC  could 
improve  implementation  of  AFI  63-124  Air  Force-wide  (and  thus  improve 
its  ability  to  meet  the  PBSA  goal  set  by  Dr.  Gansler)  by  clarifying  how  it 
can  and  should  be  applied  to  the  diverse  types  of  services  contracts  found 
within  the  systems  side  of  AFMC. 


8Ms.  Druyun  is  the  Principal  Deputy  Assistant  Secretary  of  the  Air  Force  for  Acquisition 
and  Management.  In  May  1995,  Ms.  Druyun  issued  the  first  of  a  series  of  initiatives  she 
called  Lightning  Bolts  to  reform  Air  Force  acquisition  and  sustainment  processes.  The 
goal  was  to  implement  faster,  better,  and  cheaper  ways  of  doing  business. 
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2.  NATURE  OF  SERVICES  AT  THE  ALC 
AND  PRODUCT  CENTER 


Outline 


•  Nature  of  Services  at  the  ALC  and 
Product  Center 

•  Application  of  Performance-Based 
Practices 

•  Summary  and  Implications 
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The  remainder  of  this  Documented  Briefing  is  organized  as  follows.  First, 
we  describe  the  kinds  of  services  purchased  by  the  ALC  and  Product 
Center,  focusing  on  those  characteristics  that  are  important  for 
implementation  of  AFI  63-124.  We  then  compare  the  services  acquisition 
practices  at  these  Centers  to  the  four-part  definition  of  a  performance- 
based  service  contract  found  in  FAR  Part  37.  We  conclude  with  a 
summary  of  our  observations  and  implications  for  our  current  research. 

In  the  Appendix,  we  provide  a  detailed  discussion  of  the  uncertainty 
surrounding  the  set  of  services  to  which  AFI  63-124  applies. 
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The  ALC  and  Product  Center  Purchase 
a  Broad  Range  of  Services 


•  Operational  contracting  for  installation  support  services 

•  Facilities  and  equipment  support,  information 
management,  communications,  base  supply 

•  Aircraft-related  services,  such  as  launch  and  recovery, 
air  show  support,  maintenance  of  support  equipment 
and  towing  vehicles 

•  Systems  contracting 

•  Advisory  and  assistance  services  (A&AS) 

•  Other  engineering  services 

•  Component  repair 

•  Research  and  development 

•  Logistics/sustainment  services 

RAND  Project  AIR  FORCE 


During  our  visits  to  the  ALC  and  Product  Center,  we  learned  about  two 
categories  of  services:  services  purchased  by  operational  contracting  to 
support  infrastructure  and  activities  at  the  installations  themselves,  and 
services  purchased  by  the  Product  Directorates  and  Program  Offices  to 
acquire  and  support  weapon  systems  and  other  mission  capabilities. 

The  operational  contracting  organization  at  each  Center  purchases  a  wide 
range  of  services.  Examples  from  our  discussions  are  listed  in  the  chart 
above. 

We  learned  about  five  types  of  services  on  the  systems  sides  of  these  two 
Centers. 

At  both  Centers,  personnel  focused  much  of  our  conversations  on 
advisory  and  assistance  services  (A&AS)  because  personnel  perceive  that 
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these  present  special  challenges  in  implementing  AFI  63-124.9  As  the 
name  suggests,  the  primary  purpose  of  A&AS  is  to  provide  advice  or 
assistance  in  managing  some  aspect  of  Air  Force  work.10  At  these  two 
Centers,  these  services  appear  primarily  to  include  ongoing  staff  support 
activities  in  the  Product  Directorates  and  Program  Offices,  such  as 
collection  and  analysis  of  data  from  the  field,  assistance  with  preparation 
of  briefings,  graphics,  financial  management,  administrative  support, 
technical  order  management,  and  software  support.  At  the  Product 
Center,  in  one  Program  Office  that  is  in  the  middle  of  an  acquisition,  25  of 
the  29  people  in  the  office  were  A&AS  contractor  staff. 

A  second  service  category  that  received  a  lot  of  attention  during  our 
interviews  is  engineering  services  that  fall  outside  of  A&AS.  Many 
services  that  we  learned  about  at  the  ALC  fall  within  the  subcategory  of 
"Sustaining  Engineering."  Personnel  frequently  described  them  as 
services  to  address  a  specific  problem.  For  example,  a  contractor  might  be 
hired  to  help  the  ALC  address  a  new  airframe  problem  discovered  during 
heavy  maintenance  (i.e.,  overhaul  activities)  and  to  provide  a  technical 
report  describing  (a)  its  cause,  (b)  whether  there  is  reason  to  believe  that 
the  whole  fleet  is  affected  by  it,  and  (c)  several  alternatives  to  fix  it.  Other 
examples  include  evaluation  of  a  software  problem,  updating  scenarios  in 
a  simulator,  management  of  obsolete  parts  for  a  weapon  system,  and 
analysis  of  maintenance  policy  questions  (e.g.,  the  benefits  of  two-level 
versus  three-level  maintenance11). 

A  third  type  of  service  purchased  by  the  ALC  is  component  repair.  For 
example,  the  ALC  sends  a  contractor  a  broken  line  replaceable  unit  (LRU) 
and  the  contractor  returns  a  serviceable  one  within  a  specified  number  of 
days. 

Fourth,  at  the  Product  Center,  we  learned  about  research  and 
development  services.  According  to  one  person  we  interviewed,  these 
services  generally  result  in  the  design  for  a  weapon  system  or  mission 


9Because  of  the  potential  to  use  A&AS  contractors  inappropriately  to  perform  activities 
that  should  be  performed  by  government  personnel,  A&AS  receives  high-level  attention 
and  scrutiny.  Challenges  associated  with  defining  requirements  and  managing  these 
contracts  are  the  subject  of  a  recent  DoD  Inspector  General  report.  See  Office  of  the 
Inspector  General,  Department  of  Defense  (2000). 

WAFMC  Advisory  and  Assistance  (A&AS)  Guide  (2001). 

11For  systems  supported  through  two-level  maintenance,  repairs  occur  at  the  flight  line 
and  a  centralized  facility  such  as  a  depot.  Three-level  maintenance  adds  intermediate 
maintenance  capabilities,  typically  in  a  back  shop  at  the  installation. 
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capability.  Services  like  these  are  characterized  by  a  series  of  tasks  along  a 
"critical  path"  with  associated  milestones. 

Finally,  both  the  ALC  and  Product  Center  purchase  services  that  we  have 
grouped  into  the  category  of  logistics  or  sustainment  services.  Some 
aspects  of  aircraft  heavy  maintenance  or  engine  overhaul  would  fall 
within  this  category,  although  these  can  also  be  thought  of  as  a  type  of 
remanufacturing.  Another  example  is  depot-level  support  for  a  mission- 
critical  computer  system.  The  contractor  provides  serviceable  electronic 
cards,  fixes  software  problems,  performs  diagnostics,  and  forecasts  future 
availability  problems. 

In  the  next  two  charts,  we  will  describe  the  characteristics  of  these  services 
that  are  important  for  implementation  of  AFI  63-124  by  contrasting 
operational  and  systems  services. 
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Many  Installation  Support  Services 
Are  Ideally  Suited  to  AFI  63-124 


•  Many  services  are  found  in  the  commercial 
sector 

•  Commercial  performance  standards  are  often 
available 

•  Many  services  have  a  robust  industrial  base 

•  Performance  of  many  services  can  be  evaluated 
repeatedly 

•  Performance  can  often  be  measured  objectively 

RAND  Project  AIR  FORCE 


The  Air  Force  is  in  the  process  of  incorporating  into  AFI  63-124  detailed 
examples  of  how  to  apply  performance-based  practices  within  contracts 
for  operational  services.  Given  the  nature  of  these  services,  it  can  be  fairly 
straightforward  to  apply  PBSA  to  them. 

First,  many  operational  services  have  close,  if  not  exact,  analogues  in  the 
commercial  sector.  For  example,  commercial  firms  maintain  office 
buildings,  fleets  of  vehicles,  and  information  technology  infrastructure. 
Providers  of  aircraft-related  services  must  also  maintain  the  associated 
infrastructure. 

One  person  in  operational  contracting  shared  with  us  that  the  commercial 
nature  of  many  of  these  services  makes  it  easy  to  apply  performance- 
based  practices.  There  are  commercial  standards  of  performance  readily 
available  to  reference.  In  addition,  many  firms  provide  these  types  of 
services,  so  the  Air  Force  is  not  beholden  to  any  one  service  provider, 
which  makes  performance  incentives  more  meaningful. 

Second,  many  of  the  services  purchased  through  operational  contracting 
(e.g.,  janitorial  services,  grounds  keeping,  or  equipment  maintenance) 
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occur  continuously  or  repeatedly  over  time,  so  the  customer  can  evaluate 
contractor  performance  frequently.  One  person  described  this  as  the 
ability  to  measure  the  incremental  value  added  by  the  contractor. 

Finally,  and  perhaps  most  importantly,  performance  of  many  of  these 
activities  can  be  measured  objectively.  The  customer  can  measure  the 
height  of  the  grass,  calculate  equipment  availability,  and  check  to  make 
sure  preventive  maintenance  is  performed  on  time.  These  measured 
outcomes  can  then  be  compared  to  a  clear  standard  of  success  to 
objectively  determine  whether  the  contractor  performed  well. 
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Systems  Services  Differ  from 
Operational  Services  in  Important  Ways 


•  Some  activities  have  a  “task  order”  flavor 

•  Limited  opportunities  to  evaluate  performance 

•  It  may  be  difficult  to  define  a  “successful”  outcome  in 
advance 

•  “Learning  that  what  you  asked  for  is  impossible  may 
still  be  useful” 

•  Performance  evaluation  is  subjective  for  some  services 

•  Many  services  are  evaluated  as  “pass/fail” 

•  Competition  is  limited  in  many  cases 

•  Proprietary  data,  obsolescence,  rapid  evolution  of 
technology,  limited  demand 
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During  our  interviews,  personnel  repeatedly  offered  their  assessments  of 
how  services  on  the  systems  side  differ  from  operational  contracting 
activities.  Here,  we  focus  on  the  distinctions  that  are  relevant  to 
implementation  of  AFI  63-124. 

First,  many  activities  on  the  systems  side  have  a  "task  order"  flavor,  rather 
than  being  ongoing  activities  that  occur  repeatedly  over  time.  As 
discussed  earlier,  engineering  support  services  often  address  a  specific 
problem.  Repair  of  certain  types  of  components  may  occur  infrequently. 
Research  and  development  activities  can  have  long  stretches  between 
critical  milestones.  In  each  of  these  cases,  the  customer  has  limited 
opportunities  to  evaluate  performance.  Although  contractors  often 
provide  monthly  reports  on  their  activities,  more  than  one  person  asserted 
that  these  are  only  loosely  correlated  with  whether  the  Air  Force  gets  what 
it  needs.  One  person  noted  that  progress  within  engineering  and  research 
and  development  services  does  not  occur  in  a  linear  fashion;  there  may  be 
periods  during  which  little  progress  is  made  interspersed  with  others 
during  which  much  is  accomplished. 
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Next,  it  may  be  quite  difficult  to  define  a  successful  outcome  ex  ante.  For 
example,  the  new  desired  capability  might  not  be  feasible  given  current 
technology  or  funding  levels.  Personnel  at  the  ALC  described  an 
engineering  services  contract  to  analyze  more  efficient  ways  to  track  tool 
usage  by  aircraft  maintainers  in  the  depot.  At  the  end  of  the  study,  each 
of  the  alternatives  proposed  by  the  contractor  required  more  resources 
than  the  ALC  could  provide;  however,  personnel  from  the  Product 
Directorate  that  paid  for  the  study  indicated  that  they  were  pleased  with 
the  contractor's  efforts  and  the  information  they  received.  In  their  eyes, 
actually  implementing  a  new  way  to  track  tool  usage  was  not  the  only 
possible  successful  outcome  for  the  study.  One  of  the  Program  Offices 
described  a  depot-level  maintenance  contract  for  a  mission-critical 
computer  system.  It  would  seem  straightforward  to  determine  success  for 
this  type  of  contract — i.e.,  a  specified  level  of  system  availability — but  a 
lack  of  funding  caused  the  Program  Office  to  constantly  prioritize  the 
contractor's  maintenance  activities  to  address  only  the  most  critical  issues 
at  any  one  time.12 

In  part  due  to  the  difficulty  in  defining  success  ex  ante,  personnel 
expressed  the  opinion  that  many  services  on  the  systems  side  cannot  be 
evaluated  objectively.  In  fact,  personnel  repeatedly  described  outcomes  as 
"pass/ fail" — i.e.,  either  the  result  of  the  service  met  the  customer's  needs 
or  it  didn't.  A&AS  contractors  "pass"  if  they  satisfy  the  needs  of  the 
Program  Offices  or  Product  Directorates  that  they  support.  An 
engineering  assessment  of  how  to  fix  a  new  maintenance  problem 
succeeds  if  Product  Directorate  engineers  are  satisfied  that  the  potential 
solutions  make  sense.  A  research  and  development  project  is  successful  if 
the  Program  Office  feels  that  the  contractor  made  a  good  effort  to  figure 
out  how  to  provide  the  desired  capability,  even  if  the  capability  proves  to 
be  too  difficult  at  this  time. 

Finally,  our  discussions  indicate  that  competition  is  more  limited  for 
systems  services  contracts  than  for  operational  services  contracts.  Many 
of  the  systems  contracts  we  learned  about  during  our  two  visits  are  sole- 


12Similarly,  because  of  low  turnover  of  inventories  of  some  Air  Force  components,  it  may 
be  difficult  to  assess  whether  some  repair  contracts  are  successful.  An  Air  Force 
customer  may  not  discover  a  problem  with  a  given  repair  until  after  the  contract  that 
covered  that  repair  has  been  closed  out. 
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source  contracts.13  This  can  occur  for  a  variety  of  reasons.  For  some 
maintenance  contracts,  the  Air  Force  does  not  own  the  technical  data,  so 
only  the  original  equipment  manufacturer  (OEM)  can  provide  the  service. 
Personnel  at  both  Centers  said  that  technological  obsolescence  also 
contributes  to  the  prevalence  of  sole-source  contracts.  Some  of  the 
electronic  components  on  Air  Force  weapon  systems  are  based  on  older 
technologies  that  are  no  longer  used  in  the  commercial  sector.  This  means 
that  sometimes  even  the  OEMs  are  no  longer  willing  or  able  to  support 
older  military-unique  technologies,  so  another  firm  must  be  hired  to 
reverse-engineer  parts  of  weapon  systems.  With  limited  demand  for  the 
capability  to  repair  or  enhance  such  systems,  the  Air  Force  often  cannot 
support  more  than  one  source  of  supply.  In  other  mission  areas,  the  Air 
Force  is  constantly  pushing  the  technical  community  to  come  up  with 
better  ways  to  meet  its  needs.  For  military-unique  equipment.  Air  Force 
personnel  told  us  that  it  is  difficult  to  generate  enough  demand  to  support 
the  efforts  of  more  than  one  firm  to  develop  specialized  expertise  and 
invest  in  pushing  technology  forward.  This  is  especially  problematic  for 
some  of  the  newer  aircraft  weapon  systems  with  small  fleet  sizes. 


13Based  on  an  analysis  of  FY01  DD350  data  on  Air  Force  service  contracts,  our  colleague, 
Mary  Chenoweth,  notes  that  AFMC's  Product  Centers  allocated  56  percent  of  their 
systems  service  contract  dollars  (captured  in  the  DD350  data)  through  sole-source 
contracts;  the  figure  was  46  percent  for  AFMC's  ALCs.  In  comparison,  29  percent  of 
operational  services  contract  dollars  were  awarded  through  sole-source  contracts. 
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The  ALC  and  Product  Center  Question 
Which  Services  Are  Subject  to  PBSA 


•  Regulations  provide  different  definitions  of 
“services”  and  different  lists  of  PBSA 
exemptions 

•  “Repair”  versus  “remanufacturing”  is  an  issue 
at  the  ALC 

•  The  Product  Center  hesitates  to  call  activities 
on  the  systems  side  “services” 

•  Some  seem  to  think  PBSA  should  only  apply  to 
services  governed  by  the  Service  Contract  Act 
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During  our  interviews  at  both  Centers,  we  were  repeatedly  asked  how  we 
define  services  for  the  purpose  of  our  research.  This  is  a  central 
implementation  question  because,  rather  than  listing  every  activity  for 
which  performance-based  practices  apply,  AFI  63-124  simply  states  that  it 
applies  to  all  service  contracts  greater  than  $100,000  annually,  except  for  a 
list  of  exemptions  found  in  Attachment  2  of  the  AFI.  Thus,  one  must 
understand  the  definition  of  the  universe  of  services  to  determine  whether 
the  AFI  applies  to  a  given  activity. 

We  discovered  that  the  reason  the  answer  to  the  service  question  was  so 
important  to  the  people  we  interviewed  at  the  Centers  is  that  many  of 
their  activities  that  look  like  services  to  us  are  not  considered  services 
within  the  systems  side  of  AFMC.  Thus,  many  questioned  how  (if  at  all) 
AFI  63-124  applies  to  them.  The  question  of  applicability  was  reinforced 
by  the  emphasis  on  operational  contracting  activities  within  the  current 
version  of  the  AFI. 

Each  time  we  were  asked  how  we  define  services,  we  responded  with  the 
definition  found  in  FAR  Part  37.101,  which  governs  service  contracts. 
According  to  this  regulation,  a  service  contract  is  one  in  which  the 
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primary  task  of  a  contractor  is  to  perform  an  identifiable  task  rather  than 
to  furnish  an  end  item  of  supply.14 

We  quickly  learned  that  this  definition  is  not  easy  to  apply,  as  a  variety  of 
regulations  provide  different  lists  of  what  might  be  considered  a  service. 
The  legislation  most  often  cited  in  our  interviews  was  the  Service  Contract 
Act  of  1965,  which  places  certain  wage  and  employment  obligations  on  a 
contractor.  The  section  of  the  Code  of  Federal  Regulations  (29  CFR  4.130) 
that  implements  this  Act  provides  a  list  of  55  services  to  which  the  Act 
applies.  FAR  Part  22.1003-5,  which  implements  the  Act  for  federal 
contracts,  provides  a  more  concise  list  of  services,  all  of  which  are 
included  in  the  CFR  list.  FAR  Part  22  also  has  a  long  section  that 
distinguishes  repair  services,  which  are  governed  by  the  Service  Contract 
Act,  from  remanufacturing,  which  is  considered  to  fall  under  the  Walsh- 
Healey  Act  (under  which  contractor  obligations  are  slightly  different). 

In  addition  to  lists  that  help  define  what  constitutes  a  service,  there  are 
other  regulatory  lists  that  describe  what  services  should  be  purchased 
using  performance-based  practices.  For  example,  FAR  Part  37  lists  several 
service  "areas"  covering  many  services  that  are  listed  in  the  CFR  related  to 
the  Service  Contract  Act,  but  these  areas  also  include  several  services  that 
are  not  explicitly  mentioned  in  the  CFR  or  in  FAR  Part  22.  Among  these 
are  advisory  and  assistance  services  (A&AS),  architect-engineer  services, 
communications  services,  research  and  development,  and  transportation 
services.  However,  FAR  Part  37  explicitly  exempts  four  types  of  services 
from  performance-based  practices:  certain  architect-engineer  services, 
construction,  utility  services,  and  services  that  are  incidental  to  supply 
purchases.  This  list  differs  in  some  ways  from  the  exemptions  in  the 
current  version  of  AFI  63-124.  Finally,  the  Air  Force's  PBSA 
Implementation  Plan  (developed  in  response  to  Under  Secretary  of 
Defense  Gansler's  requirement  that  50  percent  of  all  services  be  acquired 
in  a  performance-based  manner  by  2005)  provides  a  "universe  of  services" 
to  which  PBSA  should  be  applied.15  This  list  contains  some,  but  not  all,  of 
the  services  in  FAR  Part  37  and  adds  base  operations  and  support 
services,  medical  services,  and  undefined  "other"  services.  Like  FAR 
Part  37,  the  Air  Force  implementation  plan  excludes  architect-engineer 


14" 'Service  contract'  means  a  contract  that  directly  engages  the  time  and  effort  of  a 
contractor  whose  primary  purpose  is  to  perform  an  identifiable  task  rather  than  to 
furnish  an  end  item  of  supply.  A  service  contract  may  be  either  a  nonpersonal  or 
personal  contract.  It  can  also  cover  services  performed  by  either  professional  or 
nonprofessional  personnel  whether  on  an  individual  or  organizational  basis." 

15See  U.S.  Air  Force  (2000). 
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services,  but  unlike  the  FAR,  it  also  excludes  research  and  development 
acquisition. 

To  further  illustrate  the  complexity  of  distinctions  among  services  and 
opinions  about  services  to  which  PBSA  applies,  many  personnel  at  the 
ALC  classify  their  component  repair  contracts  as  remanufacturing  or 
supply  activities  rather  than  services.16  As  such,  the  contracts  are  exempt 
from  the  Service  Contract  Act  (Walsh-Healey  applies  instead).  Because  of 
this  distinction,  personnel  feel  justified  in  exempting  these  contracts  from 
PBSA.  (More  detail  about  the  differences  between  these  Acts  and  their 
relevance  to  PBSA  is  provided  in  the  Appendix.)  At  the  Product  Center,  a 
senior  executive  said  that  there  was  "an  aversion  to  calling  the  process  of 
design,  development,  and  production  a  service."  In  a  discussion  about 
performance  management,  a  program  manager  pointed  out  "R&D  is  not 
under  the  Service  Contract  Act";  another  said  the  Product  Center  does  not 
consider  R&D  to  be  a  service  because  it  results  in  a  design  (i.e.,  product). 
In  another  discussion,  we  heard  "Logistics  personnel  don't  have 
experience  with  the  Service  Contract  Act,  and  fear  it  [because  it  increases 
complexity,  limits  contract  lengths17,  and  increases  wage  rates]."  These 
comments  all  arose  during  discussions  of  performance-based  practices.  In 
light  of  the  comments  from  the  ALC,  it  is  clear  that  some  organizations  at 
each  of  these  Centers  have  developed  the  impression  that  PBSA  should 
only  be  applied  to  contracts  that  involve  services  covered  by  the  Service 
Contract  Act.18 

We  suspect  that  these  comments  are  a  reflection  of  the  understanding  at 
the  ALC  and  the  Product  Center  that  PBSA  is,  in  general,  easier  to 
implement  in  operational  services  (grounds  maintenance,  etc.),  many  of 
which  can  be  accomplished  by  people  with  basic  skill  levels,  than  in 
systems  services,  which  often  require  personnel  with  specialized  training. 


16This  is  because  they  view  the  serviceable  component,  rather  than  the  repair,  as  the  end 
product. 

17Contract  length  for  contracts  subject  to  the  Service  Contract  Act  is  limited  to  five  years 
(see  41  USC  353).  Some  component  repair  contracts  cover  ten  years. 

18In  an  analysis  of  FY01  DD350  data  on  Air  Force  service  contracts,  Mary  Chenoweth 
found  that  45  percent  of  operational  service  contract  dollars  are  recorded  as  being 
associated  with  services  that  fall  under  the  Service  Contract  Act;  only  1  percent  is 
associated  with  the  Walsh-FIealey  Act.  In  contrast,  26  percent  of  ALC's  and  7  percent  of 
Product  Center's  systems  service  contract  dollars  are  associated  with  the  Service  Contract 
Act,  while  28  and  44  percent,  respectively,  fall  under  Walsh-Healey. 
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We  heard  that  the  lower  skilled  people  are  exactly  those  whom  the  Service 
Contract  Act  was  designed  to  protect.19 


19In  fact,  the  definition  of  a  “service  employee"  given  in  FAR  Part  22.1001  excludes  “any 
person  employed  in  a  bona  fide  executive,  administrative,  or  professional  capacity." 
(emphasis  ours) 
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3.  APPLICATION  OF  PERFORMANCE-BASED 

PRACTICES 


Outline 


•  Nature  of  Services  at  the  ALC  and 
Product  Center 


•  Application  of  Performance-Based 
Practices 


•  Summary  and  Implications 


RAND  Project  AIR  FORCE 


We  now  turn  to  the  successes  and  challenges  personnel  at  the  ALC  and 
Product  Center  are  encountering  in  their  efforts  to  implement 
performance-based  practices  in  their  service  contracts.  We  focus  primarily 
on  systems  contracting  activities. 
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ALC  and  Product  Center  Personnel  Assert  Many 
Systems  Contracts  Meet  Three  PBSA  Requirements 


s/  1)  Describe  requirements  in  terms  of  results  required 
rather  than  the  methods  of  performance  of  the  work 

?  2)  Use  measurable  performance  standards  and  quality 
assurance  surveillance  plans 

s/  3)  Specify  procedures  for  reduction  of  fee  or  reduction 
to  price  of  a  fixed-price  contract  when  services  are 
not  performed  or  do  not  meet  contract  requirements 
•  Air  Force  encourages  reperformance  at  no  cost 
when  possible 

s/  4)  Include  performance  incentives  when  appropriate 

From  FAR  Part  37  and  AFI  63-124 

RAND  Project  AIR  FORCE  1 


According  to  FAR  Part  37.6  (which  AFI  63-124  implements),  performance- 
based  contracts  satisfy  four  criteria.20  First  and  foremost,  requirements 
must  reflect  what  the  purchaser  or  user  of  the  services  needs,  not  how  the 
work  should  be  performed. 

Second,  there  should  be  measurable  performance  standards  (for  quality, 
timeliness,  etc.)  and  performance  thresholds  so  that  the  purchaser, 
through  the  quality  assurance  surveillance  plan,  can  track  performance 
against  clear  goals. 

Third,  the  contract  should  contain  provisions  to  reduce  the  fee  or  the  cost 
of  a  fixed-price  contract  if  services  do  not  meet  the  purchaser's  specified 
needs.  The  Air  Force  considers  the  contract  clauses  52.246-4  and  52.246-5, 
which  specify  reperformance  at  no  additional  cost  in  the  event  of 
unsatisfactory  work,  to  satisfy  this  requirement. 


20For  a  contract  to  be  recorded  in  the  Federal  Procurement  Data  System  (DD  Form  350) 
as  performance-based,  a  minimum  of  80  percent  of  the  anticipated  obligations  under  the 
procurement  must  satisfy  the  four  criteria  (FPDS,  2002). 


23 


Finally,  performance  incentives,  such  as  award  fees  or  award-term 
contracts,  should  be  used  when  appropriate. 

In  most  of  our  interviews,  we  walked  through  this  definition  to  learn  how 
personnel  are  applying  these  practices  and  where  they  see  additional 
opportunities.  People  at  the  ALC  and  Product  Center  said  that  they  were 
not  meeting  all  of  the  requirements  of  AFI  63-124;  one  individual  said  that 
we  would  not  find  one  contract  on  the  systems  side  of  AFMC  that  satisfies 
all  four  criteria  of  the  instruction.  Nonetheless,  the  opinion  of  most  of  the 
people  we  talked  to  is  that  they  have  been  writing  performance-based 
contracts  for  years. 

In  reference  to  the  four-part  definition  of  a  performance-based  service 
contract,  across  the  range  of  services  contracts  discussed,  contracting 
officers  and  requirements  personnel  felt  that  the  majority  of  their 
descriptions  of  service  requirements  in  their  contracts  are  outcome-based 
(i.e.,  stated  in  terms  of  “what"  not  "how");  effective  quality  assurance 
programs  to  manage  outcomes  are  in  place;  reperformance  is  an  option  to 
address  any  deficiencies;  and  incentives  are  available. 

In  fact,  AFMC  FAR  5337.91  (Services  Contracts),  which  was  rescinded 
when  AFI  63-124  was  approved  for  AFMC,  listed  many  of  the  key 
components  of  a  performance-based  service  contract — and  at  least  one 
person  we  interviewed  thought  that  A&AS  was  better  managed  under  the 
old  regulation.21  We  were  also  told  that  when  the  old  regulation  was  in 
force,  there  were  separate  training  coordinators  for  operational 
contracting  and  A&AS  contracting. 

The  problem  that  virtually  everyone  on  the  systems  side  of  the  ALC  and 
Product  Center  has  with  implementation  of  AFI  63-124  is  the 
incorporation  of  measurable  performance  standards.  In  the  next  few 
charts,  we'll  summarize  comments  on  the  four  parts  of  a  performance- 
based  service  contract,  saving  the  difficult  part  (measurable  performance 
standards)  for  last. 


21  The  rescinded  regulation  described  "policies  and  responsibilities  for  surveillance  of 
service  contracts  where  the  place  of  performance  for  the  effort  is  the  center  or  laboratory 
where  the  contract  is  awarded."  While  the  regulation  also  said  that  the  contracting 
officer  was  required  to  "ensure  contractor  performance  is  measurable  and  appropriate 
surveillance  can  be  accomplished,"  the  AFI  63-124  language  requiring  "measurable 
performance  standards”  is  apparently  interpreted  to  be  more  burdensome,  (emphasis 
ours) 
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Describing  Results  Required  Is 
Generally  Not  a  Problem 


•  Many  requirements  can  be  stated  in  terms  of 
“what,”  not  “how” 

•  Sometimes  how  something  is  done  still  matters 

•  Environmental  management,  flight  safety, 
packaging 

•  Confusion  about  AFI  terminology  for 
description  of  requirements 

•  “A  SOO  is  performance-based” 

•  “A  SOW  is  a  contractor’s  detailed  ‘how  to’ 
plan” 

RAND  Project  AIR  FORCE 


We  were  told  that  for  many  service  contracts  at  these  two  Centers, 
describing  requirements  in  terms  of  results  is  relatively  easy.  One  A&AS 
contract  describes  a  task  as  follows: 

The  contractor  shall  assist  with  the  preparation,  production  and 
presentation  of  performance  measurements.  Such  measurements 
shall  include,  but  not  be  limited  to  [Center]  metrics,  SPO  metrics, 
and  Business  Area  metrics.  These  performance  indicators  shall  be 
maintained  and  updated  monthly. 

We  were  told  that  component  repair  contracts  use  a  lot  of  "boilerplate" 
performance-based  language  indicating  that  the  Air  Force  will  send  the 
contractor  an  unserviceable  item  and  expect  to  receive  a  serviceable  one 
within  a  specified  number  of  days.  There  are  clear  performance  metrics 
for  quality  (deficiency  reports)  and  timeliness.22  For  a  vehicles 


22Mary  Chenoweth's  analysis  of  quality  deficiency  reports  for  a  group  of  Air  Force 
component  repair  contracts  indicated  that  while  these  reports  represent  the  best  single 
source  of  data  on  contractor  quality,  their  usefulness  is  limited  by  incomplete  record¬ 
keeping. 
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maintenance  contract,  mobile  maintenance  performance  requirements  are 
specified  by  the  allowed  response  times  for  roadside  assistance  calls, 
which  vary  by  time  of  day.  One  Product  Directorate  described  a  portion 
of  its  software-related  services  as  "the  contractor  will  update  software  X  in 
equipment  Y  to  have  capability  Z." 

Most  people  we  interviewed  seemed  convinced  that  using  performance- 
based  requirements  is  beneficial;  however,  several  people  at  both  Centers 
noted  that  there  are  many  services  that  should  not  be  purchased  using 
"pure"  performance-based  requirements.  Environmental  management 
services  were  cited  more  than  once  as  an  example.  Personnel  told  us  that 
the  Air  Force  cannot  transfer  its  legal  responsibility  to  the  contractor;  as  a 
result,  it  must  have  a  greater  degree  of  control  over  how  the  services  are 
provided  to  ensure  compliance  with  federal  regulations.  Similarly, 
aircraft  maintenance  related  to  safety-of-flight  issues  must  be  performed 
according  to  a  step-by-step  process  governed  by  technical  orders. 
Contractor  field  teams  that  install  modifications  at  installations  are 
provided  with  detailed  instructions.  Some  component  repair  contracts 
specify  packing  and  shipping  requirements  in  laborious  detail.  When  we 
asked  about  this,  we  were  told  that  unlike  many  commercial  items,  some 
types  of  Air  Force  components  may  sit  in  warehouses  for  many  months  or 
years,  so  packaging  integrity  is  crucial. 

The  AFI's  reference  to  the  requirements  document  as  a  "statement  of 
work"  led  to  some  confusion  among  systems  personnel.  There  was  some 
discussion  by  the  systems  personnel  at  both  Centers  about  perceived 
differences  between  statements  of  work  (SOWs)  and  statements  of 
objective  (SOOs).  In  the  opinion  of  many,  a  SOO  is  the  document  that  tells 
the  contractor  what  the  Air  Force  needs;  the  contractor  responds  with  a 
SOW  that  describes  its  plan  to  meet  that  need.  One  person  even 
commented  that  the  AFI  63-124's  use  of  the  term  SOW  (even  though  the 
AFI  specifies  that  the  term  should  be  interpreted  broadly)  and  its 
description  of  the  SOW  format  might  actually  discourage  the  writing  of 
short  performance-based  requirements  documents.  In  this  person's 
opinion,  engineers  accustomed  to  writing  SOWs  in  the  old  style  would 
interpret  the  AFI  format  as  a  requirement  to  include  all  of  the  "how  to" 
information  they  have  used  in  the  past.23  Operational  contracting 


23For  some  component  repair  contracts,  the  Air  Force  may  include  a  SOO  in  the  request 
for  proposal  (RFP).  Then  a  contractor  will  propose  how  it  will  meet  the  objectives  in  a 
SOW.  If  acceptable,  the  contract  then  includes  the  SOW  that  the  contractor,  not  the  Air 
Force,  produced. 


26 


personnel  did  not  express  as  much  confusion  over  the  distinction  between 
SOWs  and  SOOs. 


As  a  side  observation,  we  interviewed  several  people  who  referred  to 
themselves  as  performance-based  contracting  "converts."  These  people 
used  to  tell  contractors  how  to  perform  the  work  and  now  are  supporters 
of  telling  contractors  what  the  Air  Force  needs  and  letting  the  contractor 
figure  out  how  to  meet  those  needs.  A  program  manager  from  one 
Product  Directorate  said  that  he  used  to  work  with  industry  early  in  the 
process  to  help  specify  the  requirements.  Sometimes  the  collaboration 
ultimately  led  to  requirements  so  specific  that  they  could  only  be  satisfied 
by  a  particular  model  number  produced  by  the  contractor  with  whom  he 
had  conferred  during  the  requirements  development  phase.  Now  this 
program  manager  promotes  working  with  industry  and  with  contracting 
officers  to  define  the  Air  Force's  needs  independently  of  technologies 
available  by  any  particular  contractor  and  using  this  to  construct  a  SOO. 
He  asserted  that  the  performance-based  approach  has  led  to  increased 
competition  as  well  as  more  innovative  and  improved  solutions  to  the 
Product  Directorate's  needs.  Similarly,  one  requirements  person  from 
operational  contracting  said  that  he  believes  the  new  performance-based 
approach  provides  more  opportunities  for  contractors  to  succeed  in 
meeting  the  Air  Force's  needs. 
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Education  on  “How”  Versus  “What”  Is 
Still  Needed  for  People  New  to  PBSA 


•  Many  personnel  on  the  systems  side  of  the  ALC 
struggle  to  relate  SAF/AQC  briefing  to  their  job 

•  Some  Product  Center  personnel  found  DAU 
course  helpful 

•  There’s  no  substitute  for  working  through  a  real 
example 

•  Support  from  AFMIA,  AFCESA  was 
particularly  useful  for  operational  contracting 

RAND  Project  AIR  FORCE 


Even  though  personnel  reported  a  lot  of  success  in  using  performance- 
based  service  requirements,  we  learned  about  some  barriers  to  success  at 
both  Centers.  We  were  told  that  some  Program  Offices  still  try  to 
purchase  A&AS  by  specifying  requirements  for  specific  numbers  of 
people  with  certain  skills  and  experience.  In  discussions  with  personnel 
from  operational  contracting  at  both  Centers,  we  were  told  that  some 
requirements  personnel  are  having  difficulty  distinguishing  between 
specifying  "what"  they  need  and  specifying  "how"  the  work  should  be 
accomplished.  For  example,  there  was  confusion  about  whether  it  is 
appropriate  to  tell  a  painting  contractor  that  the  paint  should  be  mixed  to 
match  across  a  large  wall  or  that  the  wall  surface  should  be  prepared  for 
paint,  filling  in  cracks  and  smoothing  uneven  surfaces.  Some  asserted  that 
if  they  specify  more  than  "paint  the  wall,"  the  requirement  is  no  longer 
performance-based.  These  personnel  felt  that  they  could  not  express  what 
the  installation  truly  needed  within  the  new  format.  We  discerned  that 
one  source  of  difficulty  is  that  some  requirements  personnel  remember  the 
previous  performance  problems  that  led  to  the  old  process-based 
statements  of  work,  instructions,  and  regulations,  and  fear  that  the 
problems  will  recur  with  statements  of  work  that  are  less  detailed. 
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Based  on  these  anecdotes,  it  appears  to  us  that  the  Air  Force  could  benefit 
from  additional  training  on  how  to  implement  performance-based 
practices,  particularly  for  requirements  personnel.  Many  people  we 
interviewed  had  received  some  training  already.  Most  of  those  at  the  ALC 
had  attended  a  recent  on-site  briefing  on  PBSA  provided  by  SAF/AQC. 
However,  those  who  work  on  the  systems  side  expressed  frustration  that 
the  information  provided  was  tailored  to  operational  contracting  activities 
rather  than  the  services  they  purchase  for  the  Product  Directorates.  In 
contrast,  no  one  at  the  Product  Center  had  seen  this  briefing.  However,  a 
select  group  that  included  XP  (Plans  and  Programs)  and  acquisition 
personnel  had  attended  a  course  taught  by  the  Defense  Acquisition 
University  (DAU).  We  were  told  that  they  thought  it  was  quite  useful. 

When  we  raised  the  topic  of  training  with  the  operational  contracting 
organization  at  each  Center,  we  were  told  that  formal  training  is  useful 
but  not  sufficient,  even  when  it  is  built  around  case  studies.  They  believed 
that  the  most  effective  learning  experience  is  working  through  a  real 
example  with  knowledgeable  support.  Personnel  at  one  Center  received 
hands-on  assistance  from  the  Air  Force  Manpower  and  Innovation 
Agency  (AFMIA);  a  person  spent  a  week  with  them  helping  to  convert 
their  requirements  from  process-based  to  performance-based.  Personnel 
at  the  other  Center  benefited  from  information  provided  by  the  Air  Force 
Civil  Engineer  Support  Agency  (AFCESA).  Personnel  stated  that  they 
believed  this  was  the  best  training  that  they  had  received  on  applying 
performance-based  practices  to  their  work. 
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Reperformance  Is  More  Common 
Than  Reductions  in  Fee  or  Cost 


•  Most  use  the  reperformance  option 

•  Although  some  feel  that  it  is  inappropriate  for 
professional  services 

•  Services  appropriate  for  cost-based  contracts 
present  challenges 

•  Time  and  materials  (T&M)  makes  reductions 
difficult 

•  Failure  may  not  be  due  to  shirking,  difficult  to 
justify  reperformance  at  no  cost 

•  Contract  termination  or  poor  CPARS  ratings  are 
other  options 

RAND  Project  AIR  FORCE 


In  our  interviews,  personnel  discussed  the  use  of  formal  (contractual)  and 
informal  incentives  to  align  contractor  activities  with  the  needs  of  the  Air 
Force. 

In  our  discussions  of  negative  incentives,  we  learned  of  a  few  examples  in 
which  reductions  in  fees  or  cost  (commonly  referred  to  as  "deducts")  are 
used.  One  Product  Directorate  at  the  ALC  has  an  aircraft  heavy 
maintenance  contract  in  which  the  contractor  and  the  Air  Force  split  any 
cost  overruns  (or  underruns).24  We  heard  about  a  contract  for  contractor 
field  teams  (CFTs)  that  includes  a  negative  incentive,  although  we  were 
not  provided  any  details.  However,  many  personnel  from  both  Centers 
emphasized  a  preference  for  the  use  of  reperformance  at  no  cost  to  the  Air 
Force  instead  of  these  reductions. 

Interestingly,  one  person  said  that  he  was  uncomfortable  with  the  idea  of 
applying  reperformance  to  professional  types  of  services  such  as  A&AS, 


24There  was  some  dissatisfaction  arising  from  the  fact  that  quality  is  not  included  in  the 
incentive.  Product  Directorate  staff  felt  that  the  contractor  was  reducing  quality  to  cut 
costs  and  thus  share  in  the  savings. 
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research  and  development,  and  engineering  services.  He  felt  that 
reperformance  is  much  more  relevant  to  "workmanship"  types  of  services. 
Again,  we  interpret  this  as  related  to  the  feeling  that  AFI  63-124  is  more 
applicable  to  operational  contracting  and  Service  Contract  Act  kinds  of 
activities  than  to  the  typical  service  on  the  systems  side  of  AFMC. 

Many  personnel  indicated  that  services  for  which  the  nature  of  the  work 
(and  even  what  a  "successful"  outcome  will  look  like)  is  not  known  up¬ 
front,  such  as  some  engineering  services  and  research  and  development, 
present  their  own  challenges  in  implementing  negative  incentives.  We 
were  told  that  it  makes  sense  to  purchase  such  services  through  cost-based 
contracts,  which  personnel  felt  limited  their  ability  to  exercise  deducts 
(perhaps  the  FAR  definition  of  a  performance-based  service  contract 
focuses  on  the  use  of  negative  incentives  in  fixed-price  contracts  for  this 
reason).25  The  two  primary  cost-based  contract  types  discussed  during 
our  interviews  were  time  and  materials  (T&M)  and  cost  plus  fixed  or 
incentive  fee.26  It  appears  to  us  that  time  and  materials  contracts  are  more 
commonly  used  at  the  ALC  than  the  Product  Center.27  In  these  cases,  the 
Air  Force  does  not  have  an  opportunity  to  reduce  fee,  and  personnel 
indicated  that  it  is  difficult  to  justify  not  paying  contractor  costs.  More 
generally,  it  appears  to  us  that  personnel  at  both  Centers  find  it  difficult  to 
justify  reperformance  of  these  types  of  services  at  no  cost  due  to  the  fact 
that,  as  discussed  earlier,  the  contractor  might  not  have  met  the  original 
goal  for  some  reason  beyond  its  control.  For  example,  it  might  be 
impossible  given  current  technology  and  funding  levels  to  create  a  new 
capability  or  expand  the  capability  of  an  existing  system.  In  these  cases,  it 
is  unreasonable  to  expect  the  contractor  to  fund  continued  research  on  its 
own. 


25 In  a  firm  fixed-price  contract,  the  price  paid  for  the  service  is  not  adjusted  based  on  the 
cost  incurred  by  the  contractor  in  providing  the  service  (FAR  Part  16,  Subpart  16.202). 
26In  a  T&M  contract,  a  contractor  is  paid  for  direct  labor  hours  (a  specified  fixed  hourly 
rate  that  includes  wages,  overhead,  general  and  administrative  expenses,  and  profit)  and 
the  cost  of  materials  (which  can  include  material  handling  costs)  used  in  providing  the 
service.  In  a  cost  plus  fixed  fee  contract,  the  contractor  receives  reimbursement  for 
allowable  costs  incurred  in  providing  the  service  as  well  as  a  negotiated  fee  that  is  fixed 
at  the  beginning  of  the  contract.  In  a  cost  plus  incentive  fee  contract,  the  contractor 
receives  reimbursement  for  allowable  costs  incurred  in  providing  the  service  as  well  as  a 
fee  that  is  derived  from  the  difference  between  the  target  cost  for  the  contract  and  the 
total  allowable  costs  incurred  (FAR  Part  16,  Subparts  16.601, 16.306, 16.304,  and  16.405- 
!)• 

27Personnel  at  the  ALC  indicated  that  Ms.  Druyun  is  strongly  discouraging  the  use  of 
cost  plus  contracts. 
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When  asked  how  they  deal  with  a  contractor  that  clearly  isn't  doing  a 
good  job  in  such  situations,  many  personnel  felt  that  the  threat  of  contract 
termination  or  a  poor  Contractor  Performance  Assessment  Reporting 
System  (CPARS)  report  provided  the  desired  incentive.  Of  course,  these 
presumably  have  less  effect  on  sole-source  contractors. 
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Positive  Incentives  Are  More  Common 


•  Several  operational  and  systems  contracts 
have  award  fees 

•  Interest  in  award  term  contracts 

•  Some  question  the  applicability  of  incentives 
to  A&AS 

•  Many  feel  CPARS  is  an  effective  motivator 
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One  person  said  that  he  sees  the  Air  Force  moving  away  from  negative 
incentives  toward  positive  incentives  such  as  award  fees  and  award  term 
contracts.  His  comment  is  consistent  with  our  impressions  from  both 
Centers. 

A  large  operational  contract  that  will  soon  be  awarded  will  be  a  firm 
fixed-price  plus  award  fee  contract,  with  criteria  for  the  award  fee  based 
on  quality  and  technical  performance.  This  contract  uses  as  a  baseline  the 
costs  of  the  three  old  contracts  that  are  being  replaced;  contractors  are 
advised  not  to  compete  if  they  cannot  perform  for  less  than  the  price 
associated  with  the  previous  contracts.  The  contracting  officer  in  this  case 
emphatically  claimed  that  the  contractor  has  so  many  incentives  to 
perform  well  that  if  it  doesn't,  something  is  wrong. 

As  mentioned  earlier,  one  aircraft  heavy  maintenance  contract  that  we 
learned  about  is  a  fixed-price  target  fee  contract  in  which  cost 
overruns/ underruns  are  split  50-50  by  the  contractor  and  the  Air  Force. 

A  large  proposed  operational  contract  will  have  a  complex  award  fee 
based  on  the  contractor's  performance  along  three  dimensions: 
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management,  technical  ability,  and  cost  control.  Each  of  the  three 
categories  is  broken  down  into  subelements.  Weights  assigned  to 
elements  change  after  the  first  year,  with  management  receiving  relatively 
more  weight  during  the  initial  transition  year  (cost  receives  the  smallest 
weight  in  each  year).  Contractor  performance  within  each  subelement  is 
subjectively  evaluated  based  on  five  possible  ratings.  Subelement  ratings 
are  rolled  up  to  calculate  the  percentage  of  the  total  possible  fee  that  will 
be  awarded.  We  learned  of  similar  award  fee  structures  for  engineering 
services  and  sustainment  contracts  on  the  systems  side  of  this  Center. 

In  many  of  our  interviews,  personnel  expressed  interest  in  award  term 
contracts,  in  which  the  length  of  the  contract  is  formally  tied  to  contractor 
performance.  However,  we  only  saw  one  example  of  a  proposed  award 
term  plan.  The  proposed  structure  is  similar  to  the  award  fee  described 
above. 

Some  people  we  interviewed  believe  that  it  is  difficult  to  provide 
incentives  for  A&AS.  Comments  included  “How  do  you  evaluate 
whether  you  received  the  contractor's  best  advice?"  and  "How  do  you 
incentivize  pass/fail  activities?"  In  fact,  several  people  questioned 
whether  performance  incentives  are  even  appropriate  in  this  setting.  In 
their  opinion,  an  organization  should  expect  good  performance  under  an 
A&AS  contract.  Why  should  they  have  to  pay  more  to  get  the  contractor's 
best  advice  or  performance?  Instead,  these  personnel  suggested  that  it  is 
the  responsibility  of  contractors  to  reward  the  employees  that  perform 
particularly  well.  So,  bonuses  and  incentives  for  this  kind  of  work  should 
come  through  the  contractor,  not  directly  from  the  Air  Force. 

Some  offices  feel  that  feedback  through  CPARS  ratings  is  the  best  way  to 
provide  incentives  to  contractors  for  these  types  of  services.  Fortunately, 
there  is  a  feeling  at  both  Centers  that  these  performance  ratings  are  quite 
important  to  contractors  because  the  ratings  directly  affect  the  contractors' 
abilities  to  get  future  contracts.  As  an  example,  one  Center  purchases 
A&AS  through  a  large  umbrella  contract  that  includes  blanket  purchase 
agreements  with  a  number  of  prescreened  contractors.  An  individual 
Program  Office  with  a  requirement  for  A&AS  can  select  the  contractors 
that  it  would  like  to  provide  proposals;  full  and  open  competition  at  the 
task  order  level  is  not  necessary  as  long  as  the  Program  Office  chooses 
three  or  more  firms  to  compete.  Contractors  know  that  their  CPARS 
information  is  used  by  Program  Offices  to  help  determine  which  firms  get 
an  opportunity  to  compete  for  individual  task  orders. 
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The  anecdotes  on  this  chart  and  the  previous  one  indicate  a  wide  range  of 
opinions  on  the  use  of  incentives  and  disincentives.  The  manager  of  the 
CFT  contract  said  that  they  were  trying  to  get  away  from  using 
disincentives/ deducts,  but  they  were  finding  it  difficult  to  develop 
positive  incentives.  On  the  other  hand,  another  contracting  officer 
thought  that  incentives  were  not  as  effective  as  deducts  for  motivation. 
Finally,  a  third  group  thought  both  incentives  and  deducts  were  necessary 
in  combination  to  encourage  good  performance  from  the  contractor. 
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“Measurable  Performance  Standards” 
Is  a  Sticking  Point 


•  Many  interpret  FAR  language  to  require: 

•  Objective  performance  data,  collected  frequently 

•  The  desired  result  to  be  known  in  advance 

•  It  is  difficult  to  reconcile  this  interpretation  with  A&AS  and 
some  engineering  services 

•  Assessments  are  often  subjective 

•  Infrequent  deliverables 

•  Correct  answer  is  not  known  in  advance;  success  may 
not  even  be  possible 

•  When  measurable  performance  standards  are  available, 
still  worry  about  using  ex  ante  measures  of  success 

•  Changing  requirements 

•  Inability  to  control  all  inputs 
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Personnel  at  both  Centers  expressed  the  opinion  that  one  of  the  biggest 
challenges  associated  with  implementation  of  performance-based  service 
contracts  as  defined  by  FAR  Part  37  is  defining  measurable  performance 
standards  for  the  services  they  buy  on  the  systems  sides  of  these 
organizations.  This  seems  to  be  particularly  problematic  for  personnel 
working  with  A&AS  and  other  engineering  services  contracts. 

For  those  who  had  considered  the  applicability  of  AFI  63-124  to  the 
services  that  they  buy,  virtually  everyone  we  interviewed  perceived — 
based  on  the  examples  given  when  PBSA  was  briefed  to  them — that 
"measurable  performance  standards"  require  frequent  collection  of 
objective  performance  data  that  allows  an  ongoing  assessment  of 
performance  against  a  known  measure  of  success,  i.e.,  "measurable 
performance  standards"  means  that  you  can  evaluate  the  percentage  of 
time  the  contractor  met  the  performance  goal  during  a  certain  period  of 
time.  One  person  suggested  that  a  contract  is  only  performance-based  if 
there  is  a  metric  for  daily  value  added.  As  examples,  many  people  cited 
operational  contracts  such  as  grounds  maintenance.  In  their  minds,  it  is 
straightforward  to  specify  that  the  grass  height  should  be  maintained 
between  two  and  three  inches  and  then  to  measure  periodically  to  make 
sure  the  contractor  is  performing  satisfactorily.  This  process  results  in  a 
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metric  that  can  assess  the  percentage  of  time  that  a  requirement  is  met. 
However,  personnel  expressed  the  opinion  that  such  measures  and 
assessment  are  more  difficult  for  some  systems  services  such  as  A&AS 
and  other  engineering  services. 

Traditional  performance  assessments  for  A&AS  and  engineering  services 
are  often  subjective  in  nature.  In  addition,  it  may  be  difficult  to  collect  a 
series  of  data  points  on  performance  over  a  period  of  time.  Even  though  a 
program  manager  may  have  day-to-day  contact  with  A&AS  contract  staff, 
the  best  measurement  of  performance  might  be  a  subjective  assessment  of 
his  or  her  satisfaction  with  the  data  analysis  or  briefings  they  provide. 
Many  engineering  services  contracts  are  in  response  to  a  specific  problem. 
In  order  to  judge  the  quality  of  an  engineering  assessment  of  a  structural 
problem  in  the  fleet,  a  Product  Directorate  may  rely  on  its  experience  with 
similar  problems  in  the  past,  instead  of  on  some  quantitative  measure. 
Although  the  contractor's  plan  for  addressing  the  problem  usually 
includes  scheduled  milestones,  ultimate  success  (in  terms  of  meeting 
customer  needs)  may  not  be  correlated  with  the  contractor's  incremental 
progress.  When  trying  to  figure  out  how  to  add  a  particular  capability  to 
an  aircraft  subcomponent,  the  contractor  might  have  to  eliminate  many 
potential  solutions  before  finding  the  right  one,  so  progress  rarely  occurs 
in  a  linear  fashion. 

One  requirements  person  interprets  the  requirement  for  measurable 
performance  standards  to  mean  that  he  must  know  the  answer  to  the 
question  (outcome)  in  advance,  in  order  to  judge  whether  the  contractor 
has  performed  well,  which  doesn't  make  sense  for  A&AS  and  engineering 
support  services.  According  to  this  individual,  this  requires  that  "I've 
already  worked  out  my  solution,  and  I'll  compare  the  contractor's  solution 
to  mine.  Then  I'll  grade  them  [on  how  close  they  came  to  my  solution]." 
Another  person  went  so  far  as  to  say  "there's  no  metric  that  allows  an 
assessment  of  effectiveness"  in  A&AS. 

Personnel  at  both  Centers  told  us  that  sometimes  the  outcome  might  be 
that  the  problem  cannot  be  solved  given  current  technologies  or  resource 
constraints.  For  example,  one  Product  Directorate  asked  a  contractor  to 
design  an  electronic  component  made  from  commercial-off-the-shelf 
(COTS)  technology  to  replace  a  component  plagued  by  technological 
obsolescence.  Partway  into  the  project,  the  contractor  discovered  a 
military-unique  chip  in  the  original  component  that  could  not  be  replaced 
with  a  commercial  one;  therefore,  the  original  objective  for  the  contract 
was  not  feasible.  This  discovery,  while  not  the  desired  goal  initially,  does 
not  preclude  a  successful  outcome  for  the  contract.  In  this  case,  a 
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performance  standard  that  required  successful  problem  solution  (i.e.,  a 
new  component  made  entirely  from  COTS  technology)  would  have  been 
unrealistic  and  inappropriate. 

Personnel  on  the  systems  sides  of  the  two  Centers  noted  that  even  when 
objective  measures  of  performance  are  available,  it  might  not  be  wise  to  tie 
incentives  to  them.  The  evolving  state  of  knowledge — and  sometimes 
funding  constraints — means  that  requirements  for  services  on  the  systems 
side  can  change  over  time,  making  measures  of  success  determined  ex  ante 
irrelevant.  One  example  mentioned  in  our  interviews  was  that  priorities 
for  maintaining  a  mission-critical  computer  system  evolve  over  time  as  the 
Program  Manager  seeks  to  use  limited  sustainment  funds  in  the  most 
effective  way.  In  addition,  in  many  cases,  the  contractor  cannot  control  all 
the  inputs  that  contribute  to  ultimate  performance.  For  example,  we  were 
told  of  more  than  one  example  where  "timeliness"  would  be  a  poor 
measure  of  performance  because  of  contractor  reliance  on  government- 
furnished  equipment.  In  one  of  these  cases,  it  was  the  users'  responsibility 
to  turn  equipment  over  to  the  contractor  for  scheduled  maintenance. 

Some  users  would  hesitate  to  do  so,  since  turning  in  the  equipment  meant 
they  would  lose  some  capability  while  it  was  being  repaired,  and  this 
meant  the  contractor  would  sometimes  get  behind  in  its  maintenance.  It 
would  have  been  wrong  in  this  case  to  punish  the  contractor  for  not 
meeting  a  maintenance  schedule  when  it  was  not  his  fault.28 

Some  personnel  felt  that  customer  satisfaction  is  the  only  measurable 
outcome-oriented  performance  standard  in  some  cases.  One  contracting 
officer  told  us  that  he  has  tried  to  convince  SAF/AQC  that  customer 
satisfaction  measures  should  count  as  measurable  performance  standards, 
but  he  felt  he  was  rebuffed. 

Dr.  Gansler's  2000  memo  also  directed  the  services  to  create 
implementation  plans  to  promote  the  use  of  performance-based  practices 
in  their  services  acquisitions.  Interestingly,  the  Air  Force's  PBSA 
implementation  plan  (U.S.  Air  Force,  2000)  recognizes  the  difficulty  of 
applying  the  four-part  PBSC  definition  to  A&AS  and  allows  the  maximum 
practicable  application  of  PBSA  to  count  as  full  implementation: 


28In  many  component  repair  contracts,  the  “clock"  for  evaluating  repair  turnaround  time 
stops  if  the  contractor  is  waiting  on  materials  that  are  supplied  by  the  Air  Force  and 
starts  again  when  the  materials  arrive.  Presumably,  a  similar  arrangement  could  be 
implemented  in  these  cases  as  well. 
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The  use  of  PBSA  for  A&AS  will  have  unique  application  under  this 
plan. . . .  Contracts  for  A&AS  acquire  expertise  and  knowledge  for  a 
variety  of  tasks  that  result  in  subjective  and  intangible  outputs  and 
do  not  always  lend  themselves  to  the  application  of  measurable 
performance  objectives  and  thresholds  as  required  by  PBSA.  Due 
to  the  subjective  nature  of  A&AS,  the  methods  of  surveillance  (i.e., 
100%  and/ or  periodic  inspection,  random  sampling,  customer 
feedback)  used  by  the  Air  Force  today  for  routine  services  are  not 
always  appropriate.. . .  In  many  cases,  PBSA  is  not  suitable  for  all 
A&AS  requirements;  however,  the  use  of  PBSA  methods  should  be 
considered  and  applied  to  the  maximum  extent  practicable  when  it 
is  advantageous  (value  added)  to  the  Government.  For  purposes  of 
reporting  A&AS  under  this  plan,  limited  application  of  PBSA  will 
be  considered  and  reported  as  full  PBSA  implementation  (p.  3). 

Unfortunately,  the  people  we  spoke  with,  as  well  as  the  SAF/ AQC  person 
referenced  above  who  rejected  the  notion  of  using  customer  satisfaction  as 
a  performance  standard,  seemed  to  be  unaware  of  this  additional 
guidance  for  A&AS. 

This  official  acknowledgment  that  A&AS  are  not  amenable  to  the 
application  of  measurable  performance  standards  implies  that  SAF/ AQC 
interprets  such  standards  strictly,  and  this  strict  interpretation  has  been 
communicated  to  personnel  at  the  ALC  and  the  Product  Center.  Since 
there  are  several  other  kinds  of  services  in  addition  to  A&AS  for  which 
this  interpretation  is  problematic,  it  is  unfortunate  that  the 
implementation  plan  does  not  make  allowances  for  them  as  well. 
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Personnel  Feel  QASPs  Are  Effective 
Even  Without  “Measurable”  Standards 


•  Customers  can  communicate  to  contractors 
their  satisfaction  or  dissatisfaction  as  well 
as  underlying  reasons 

•  Some  feel  that  the  ability  to  monitor  and 
report  discrepancies  meets  the  intent  of 
requiring  “measurable  standards” 
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Despite  the  perceived  difficulty  of  developing  measurable  performance 
standards,  virtually  every  person  we  spoke  with  felt  that  their  Quality 
Assurance  Surveillance  Plan  (QASP)  allowed  them  to  adequately  monitor 
and  document  contractor  performance  so  that  the  Air  Force  received  the 
service  it  needed  at  a  reasonable  price.  They  could  easily  communicate 
their  opinion  of  performance  to  a  contractor,  as  well  as  the  underlying 
reasons  for  the  opinion,  so  that  any  performance  problems  would  be 
addressed. 

This  ability  was  primarily  due  to  close  communication  between  Air  Force 
and  contractor  personnel.  In  fact,  several  requirements  personnel  said 
that  they  communicate  informally  with  their  engineering  services 
contractors  several  times  each  day.  This  informal  communication  is 
supplemented  periodically  by  formal  Technical  Interchange  Meetings 
(TIMs).  We  heard  examples  where  quality  assurance  personnel  regularly 
check  with  Program  Managers  to  assess  contractor  performance  (i.e., 
customer  satisfaction)  in  A&AS.  In  addition,  many  reported  that  they 
have  formal  monthly  or  quarterly  meetings  between  Program 
Office /Product  Directorate  and  contractor  staff.  These  meetings  are  used 
to  document  performance  during  the  prior  period,  to  discuss  how  the 
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contractor  can  improve  performance  in  the  coming  period  to  better  meet 
its  customer's  needs,  and  to  communicate  any  unusual  requirements  that 
may  be  expected  during  the  coming  period. 

Although  many  personnel  we  interviewed  on  the  systems  sides  of  the 
ALC  and  Product  Center  have  not  been  checking  the  new  data  field  on  the 
DD  Form  350  indicating  their  contracts  meet  the  requirements  for  PBSA, 
two  organizations  that  described  their  successful  monitoring  programs 
said  that  they  did  record  their  contracts  as  PBSA.  In  their  opinion,  the  fact 
that  an  effective  monitoring  program  exists  satisfies  the  FAR  requirement 
of  having  measurable  performance  standards  for  a  performance-based 
service  contract.  In  these  cases,  based  on  the  Air  Force's  PBSA 
implementation  plan,  checking  the  PBSA  data  field  for  A&AS  appears  to 
be  the  right  thing  to  do. 
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4.  SUMMARY  AND  IMPLICATIONS 


Outline 


•  Nature  of  Services  at  the  ALC  and 
Product  Center 

•  Application  of  Performance-Based 
Practices 


•  Summary  and  Implications 
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We  conclude  with  a  summary  of  our  observations  from  our  ALC  and 
Product  Center  visits,  suggestions  for  steps  SAF/AQC  can  take  to 
improve  implementation  of  AFI  63-124  by  clarifying  how  it  applies  to 
systems  service  contracts,  and  implications  for  our  research. 
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Summary  of  Observations 
and  Implications 


•  It  appears  to  us  that  many  of  the  two 
Centers’  service  acquisition  activities 
already  satisfy  the  intent  of  the  FAR 
definition  of  PBSC 

•  Clarification  of  how  to  apply  AFI  63-124  to 
systems  service  contracts  is  needed 

•  Universe  of  services 

•  Name  for  the  requirements  document 

•  Examples  of  measurable  performance 
standards 
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To  summarize,  we  came  away  from  our  interviews  at  the  ALC  and 
Product  Center  with  the  impression  that  many  of  their  service  contracts 
already  incorporate  the  kinds  of  performance-based  practices  that  the  Air 
Force,  DoD,  and  federal  government  are  promoting,  even  if  some  of  these 
practices  do  not  match  a  strict  interpretation  of  the  definition  of  a 
performance-based  service  contract  in  FAR  Part  37.  However,  it  is  clear 
that  many  of  the  people  we  interviewed  on  the  systems  sides  of  these  two 
Centers  do  not  believe  their  service  contracts  can  be  classified  as  such,  so 
they  are  not  checking  the  PBSA  box  on  the  DD350  form. 

AFMC's  ALCs  and  Product  Centers  account  for  most  of  the  dollars  the  Air 
Force  spends  on  services.  SAF/AQC  can  improve  implementation  of  AFI 
63-124  and  thus  the  Air  Force's  ability  to  meet  the  PBSA  goal  set  by  Dr. 
Gansler  by  clarifying  how  the  FAR  Part  37  definition  of  a  performance- 
based  service  contract  should  be  applied  to  service  acquisition  activities 
on  the  systems  side  of  AFMC,  recognizing  the  enormous  diversity  across 
activities  and  the  implications  of  that  diversity  for  performance  evaluation 
and  management.  In  particular. 
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•  Clearer  guidance  is  needed  about  which  services  should  be 
purchased  using  the  performance-based  practices  outlined  in  the 
FAR.  For  example,  does  AFI  63-124  apply  to  contracts  written 
outside  of  the  Service  Contract  Act? 

•  Confusion  over  SOW /SOO  semantics  needs  to  be  eliminated.  Is 
there  a  generic  term  that  can  be  used  to  describe  what  the  Air  Force 
is  looking  for  in  a  performance-based  requirements  document? 

•  Performance  measurement  and  management  activities  that  satisfy 
the  requirement  for  measurable  performance  standards  need  to  be 
better  defined.  Is  the  strict  interpretation  expressed  during  many  of 
our  interviews  the  correct  one?  Does  the  ability  to  determine 
whether  the  contractor  is  meeting  the  Air  Force's  needs,  convey 
feedback  about  performance  to  the  contractor,  and  address  any 
problems  satisfy  the  requirement  for  measurable  performance 
standards? 

For  the  last  set  of  questions,  the  Air  Force  may  want  to  expand  the 
discussion  of  measurable  performance  standards  found  in  its 
implementation  plan  to  explicitly  compare  appropriate  kinds  of 
surveillance  activities  across  the  diverse  types  of  services  that  it  purchases. 
In  addition,  it  may  want  to  expand  the  "unique  application"  of  PBSA  to 
include  other  systems  services  that  have  characteristics  similar  to  A&AS. 
Finally,  the  Air  Force  should  consider  including  an  explicit  discussion  of 
this  issue  in  the  next  version  of  AFI  63-124. 
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Will  We  See  Better  Outcomes  on  the 
Systems  Side  of  AFMC  as  a  Result  of  PBSA? 


•  Sole-source  situations  present  challenges 

•  Practices  and  incentives  must  be  tailored  to 
specific  circumstances 

•  Some  types  of  incentives  may  be  ineffective 

-  Award  term  contracts 

-  CPARS 

•  May  be  difficult  to  create  baseline  from 
historical  data 

•  Application  of  performance-based  practices 
predates  AFI  63-124 
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In  our  current  research,  we  are  developing  tools  or  a  methodology  to 
assess  how  implementation  of  performance-based  practices  is  affecting  the 
Air  Force,  particularly  how  costs,  quality  of  life,  and  mission  capabilities 
have  changed  as  a  result  of  PBSA.  During  our  interviews  at  the  ALC  and 
Product  Center,  we  became  aware  of  two  issues  that  we  believe  may  affect 
our  ability  to  observe  benefits  within  service  acquisition  activities  on  the 
systems  side  of  AFMC. 

First,  as  discussed  earlier,  many  services  are  purchased  within  sole-source 
environments.  Due  to  the  lack  of  competition,  the  Air  Force  may  have  to 
tailor  its  use  of  performance-based  practices  in  order  to  see  the  same 
magnitude  of  benefits  that  they  would  expect  in  a  competitive 
environment.  For  example,  personnel  at  one  Center  reported  that  they  are 
encouraged  to  use  award  term  contracts;  however,  several  questioned 
whether  award  term  contracts  make  sense  within  the  sole-source 
environment.  They  noted  that  the  sole-source  firm  knows  it  will  be 
awarded  the  follow-on  contract  unless  the  Air  Force  stops  using  the 
relevant  system.  So  they  questioned  why  they  should  expect  such  a  firm 
to  expend  resources  to  improve  performance  if  a  longer  contract  is  the 
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only  benefit.  Similarly,  for  firms  that  predominantly  do  sole-source  work, 
the  threat  of  a  poor  CPARS  rating  may  carry  less  weight  than  it  would  for 
a  firm  that  operates  in  a  competitive  environment.  However,  other 
options  such  as  award  fees  and  deducts  were  raised  as  effective 
alternatives  in  these  cases. 

Second,  it  may  be  difficult  to  establish  a  baseline  of  contract  performance 
before  the  implementation  of  performance-based  practices.  Those  we 
interviewed  on  the  systems  side  of  the  ALC  and  Product  Center  said  that 
their  motivation  for  using  performance-based  practices  was  the 
introduction  of  acquisition  reform  in  the  mid-1990s,  which  predates  AFI 
63-124.  It  is  unclear  whether  the  data  are  available  to  study  the  impact  of 
this  change  in  contracting  practices  during  this  period. 
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APPENDIX:  WHAT  IS  A  SERVICE? 


What  Is  a  Service? 


•  General  definition 

•  Definition  for  PBSA 
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In  this  appendix,  we  discuss  the  primary  sources  of  information  on  the 
kinds  of  activities  that  should  be  considered  services.  We  then  examine 
current  guidance  on  the  set  of  services  to  which  PBSA  should  be  applied. 
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What  Is  a  Service? 

Several  Sources  Address  This  Question 


•  Service  Contract  Act  (SCA)  of  1965  (as 
amended) 

•  41  use,  351 

•  Walsh-Healey  Public  Contracts  Act 

•  41  use,  45-35 

•  FAR  Part  22,  “Application  of  Labor  Laws  to 
Government  Acquisitions” 

•  FAR  Part  37,  “Service  Contracting” 

•  OMB  Circular  No.  A-11,  Section  83 
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At  both  the  ALC  and  the  Product  Center  we  visited,  we  were  asked  how 
we  define  services.  There  are  several  important  sources  of  information 
related  to  this  question.  The  most  important  ones  are  listed  on  the  chart 
above. 

The  Walsh-Healey  Public  Contracts  Act  "is  intended  to  impose  fair  social 
employment  standards  on  government  contractors."  The  Service  Contract 
Act  (SCA)  applies  to  those  contracts  with  the  principal  purpose  of 
furnishing  services.  Both  Acts  require  payment  of  minimum  wages, 
specify  penalties  for  noncompliance,  and  restrict  employment  of  children 
and  convicts.  In  addition,  both  use  the  Fair  Labor  Standards  Act  as  a 
reference  point. 

FAR  Part  22  describes  how  both  Acts  (as  well  as  other  labor  laws)  apply  to 
government  acquisitions,  and  provides  detailed  information  for 
contracting  officers  on  applicability,  exemptions,  and  procedures  to 
implement  the  requirements  of  the  Acts. 

FAR  Part  37,  which  governs  service  contracting,  discusses  "areas  in  which 
service  contracts  are  found."  An  accompanying  list  includes  many  areas 
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that  overlap  lists  in  the  SCA  and  FAR  Part  22,  but  adds  areas  that  are  not 
discussed  in  the  legislation. 

The  purpose  of  OMB  Circular  A-ll,  Section  83  is  to  categorize  types  of 
services  by  "object  class"  for  budgeting  purposes,  since  by  law  the 
President's  budget  must  present  obligations  by  these  object  classes.  This 
document  was  most  discussed  at  the  ALC,  since  it,  along  with  additional 
guidance  from  AFMC,  was  important  in  distinguishing  A&AS  from  other 
types  of  engineering  services.  This  distinction  is  an  important  issue  in 
budgeting  at  the  ALC. 
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SCA  and  Walsh-Healey  Apply  to  Different 
Contracts  and  Have  Different  Requirements 


•  SCA  applies  to  all  contracts  in  excess  of 
$2,500,  “the  principal  purpose  of  which  is  to 
furnish  services  through  the  use  of  service 
employees” 

•  Walsh-Healey  applies  to  contracts  exceeding 
$10,000  that  are  for  the  manufacture  or 
furnishing  of  materials 

•  Both  require  maintenance  of  employee 
records,  but  requirements  differ 
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The  SCA  applies  to  contracts  over  $2,500  related  to  the  provision  of 
services.  The  Walsh-Healey  Act  addresses  contracts  over  $10,000  related 
to  the  manufacture  or  furnishing  of  materials.  Contracts  in  which 
personal  services  are  to  be  performed  in  conjunction  with  furnishing 
materials  or  equipment  are  covered  by  the  Walsh-Healey  Act,  but  if  the 
primary  purpose  is  to  provide  services,  the  SCA  applies. 

Both  the  SCA  and  Walsh-Healey  Act  require  the  maintenance  of  certain 
records.  Among  these  are: 

1.  Basic  employee  data  to  include  name,  address,  social  security 
number,  gender,  date  of  birth,  occupation,  and  job  classification. 
The  Walsh-Healey  Act  requires  that  current  work  permits  for 
minors  be  retained. 

2.  Compensation  records  to  include  amounts  and  dates  of  actual 
payment,  period  of  service  covered,  daily  and  weekly  hours, 
straight  time  and  overtime  hours  pay,  fringe  benefits  paid,  and 
deductions  and  additions.  The  Walsh-Healey  Act  also  requires  the 
retention  of  data  with  respect  to  job-related  injuries  and  illnesses. 
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specifically  logs  with  dates  and  summaries,  and  details  of 
accidents. 

The  extra  requirements  for  the  Walsh-Healey  Act  (records  that  must  be 
maintained  by  the  employer)  may  explain  the  comment  we  heard  that 
"Walsh-Healey  places  the  burden  on  the  contractor;  with  the  SC  A,  the 
government  shares  the  burden." 
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The  Code  of  Federal  Regulations 
Provides  More  Details  on  Services 


•  Types  of  service  contracts  are  “too 
numerous  to  permit  an  exhaustive  listing” 

•  Code  of  Federal  Regulations  lists  55  types 
of  services,  from  aerial  spraying  to 
warehousing 
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According  to  29  CFR  4.130,  which  implements  the  Service  Contract  Act, 

The  types  of  contracts,  the  principal  purpose  of  which  is  to  furnish 
services  through  the  use  of  service  employees,  are  too  numerous 
and  varied  to  permit  an  exhaustive  listing.  The  following  list  is 
illustrative,  however,  of  the  types  of  services  called  for  by  such 
contracts  that  have  been  found  to  come  within  the  coverage  of 
the  Act. 

The  Code  then  lists  55  types  of  services,  ranging  from  aerial  spraying  to 
warehousing  or  storage. 
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FAR  Part  22  Confuses  Things  a  Little 


•  Provides  an  unalphabetized  list 

•  All  items  (about  24)  are  included  in  29  CFR 
4.130 

•  FAR  notes  that  while  SCA  covers  repair, 
remanufacturing  (including  overhaul,  major 
modifications)  comes  under  Walsh-Healey 
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As  noted  earlier,  FAR  Part  22  implements  the  SCA  and  the  Walsh-Healey 
Act  for  government  acquisitions.  Unfortunately,  it  helps  confuse  the  issue 
of  what  constitutes  a  service.  FAR  Part  22.1003-5  provides  a  list  of  types 
of  services  "that  have  been  found  to  be  covered  by"  the  SCA.  Although  it 
notes  that  the  list  is  not  definitive  or  exclusive,  it  is  presented  as  an 
unalphabetized  list  in  a  way  that  makes  it  difficult  to  quickly  compare 
with  the  list  in  the  CFR.  All  the  items  in  the  list,  however,  are  included  in 
the  CFR  list,  and  there  are  no  new  services  listed. 

This  regulation  also  addresses  the  distinction  between  repair,  which  is 
explicitly  listed  in  the  CFR  as  an  example  of  a  service  covered  by  the  SCA, 
and  "remanufacturing."  Part  22.1003-6  is  a  long  section  that  provides 
detailed  criteria  to  determine  when  "remanufacturing"  is  to  be  deemed 
"manufacturing"  instead  of  "repair."  In  general,  "[c]  on  tracts  principally 
for  remanufacturing  of  equipment  which  is  so  extensive  as  to  be 
equivalent  to  manufacturing"  are  subject  to  the  Walsh-Healey  Act  instead 
of  the  SCA;  the  regulation  devotes  several  paragraphs  to  detailed  criteria 
that  must  be  used  to  determine  when  this  is  the  case. 
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For  example,  in  order  for  a  "major  modification"  to  be  considered 
remanufacturing,  the  following  criteria  must  be  met:  the  item  must  be 
completely  or  substantially  torn  down;  outmoded  parts  must  be  replaced; 
the  item  or  equipment  is  rebuilt  or  reassembled;  the  contract  work  results 
in  the  furnishing  of  a  substantially  modified  item  in  a  usable  and 
serviceable  condition;  the  work  is  performed  in  a  facility  owned  and 
operated  by  the  contractor. 

This  distinction  was  important  for  some  of  those  we  talked  to  at  the  ALC. 
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FAR  Part  37  Confuses  Things 
a  Little  More 


•  “‘Service  contract’  means  a  contract  that 
directly  engages  the  time  and  effort  of  a 
contractor  whose  primary  purpose  is  to 
perform  an  identifiable  task  rather  than  to 
furnish  an  end  item  of  supply.” 

•  Provides  a  list  of  several  “areas  in  which 
service  contracts  are  found” 

•  Includes  several  services  not  found  in  SCA 
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FAR  Part  37,  which  governs  service  contracts,  confuses  the  issue  even 
more.  It  provides  a  concise  definition  of  what  a  service  contract  is  (one  in 
which  "the  primary  task  of  a  contractor  is  to  perform  an  identifiable  task 
rather  than  to  furnish  an  end  item  of  supply")  and  then  lists  "areas"  in 
which  service  contracts  are  found.  This  list  covers  many  services  that  are 
listed  in  the  CFR  related  to  the  Service  Contract  Act,  but  also  includes 
several  services  that  are  not  explicitly  mentioned  in  the  CFR  or  in  FAR 
Part  22.  Among  these  are  advisory  and  assistance  services  (A&AS), 
architect-engineer  services,  communications  services,  research  and 
development,  and  transportation  services. 


55 


A  Separate  Question:  To  What 
Services  Does  PBSA  Apply? 


•  FAR  Part  37  excludes  architect-engineer 
services,  construction,  utility  services,  and 
some  others 

•  AFI  63-124  and  the  Air  Force  PBSA 
Implementation  Plan  exclude  other  services 
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There  is  lots  of  information  about  what  constitutes  a  service,  but  when 
performance-based  practices  must  be  applied  to  a  service  contract  is  a 
separate  question. 

FAR  Part  37  says  that  agencies  will  use  performance-based  contracting 
methods  to  the  maximum  extent  practicable  for  the  acquisition  of  services 
"including  those  acquired  under  supply  contracts."  The  implication  is  that 
performance-based  contracting  should  be  used  for  all  the  service  "areas" 
listed  in  FAR  Part  37.101.  However,  four  types  of  services  are  exempt  from 
performance-based  practices:  (1)  architect-engineer  services  acquired  in 
accordance  with  40  USC  541-544  as  amended;29  (2)  construction  (which 
is  covered  in  FAR  Part  36);  (3)  utility  services;  and  (4)  services  that  are 
incidental  to  supply  purchases. 


29The  relevant  regulation  is  FAR  Part  36.601-1:  "The  Government  shall  publicly 
announce  all  requirements  for  architect-engineer  services  and  negotiate  contracts  for 
these  services  based  on  the  demonstrated  competence  and  qualifications  of  prospective 
contractors  to  perform  the  services  at  fair  and  reasonable  prices.  (See  Pub. L. 92-582,  as 
amended;  40  U.S.C.  541-544.)"  However,  this  part  of  the  regulation  doesn't  make  clear 
why  these  architect-engineer  services  are  exempt. 
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The  current  version  of  AFI  63-124  applies  to  "all  service  contracts  over 
$100,000,"  but  the  instruction  also  provides  a  list  of  services  that  are 
exempt  from  PBS  A.  The  list  of  exemptions  includes,  perhaps  redundantly, 
architect-engineer  services,  but  also  includes  other  exemptions  not  shown 
in  FAR  Part  37. 

The  Air  Force's  PBSA  Implementation  Plan  (developed  in  response  to 
Under  Secretary  of  Defense  Gansler's  requirement  that  50  percent  of  all 
services  be  acquired  in  a  performance-based  manner  by  2005)  provides  a 
"universe  of  services"  to  which  PBSA  should  be  applied.  This  list  contains 
some,  but  not  all,  of  the  services  that  are  in  FAR  Part  37,  but  also  adds 
base  operations  and  support  services,  medical  services,  and  undefined 
"other"  services.  The  plan  (again  redundantly)  excludes  architect- 
engineer  services,  but  unlike  the  FAR,  also  excludes  research  and 
development  acquisition. 
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Summary 


•  Lots  of  information  on  what  constitutes  a  service 

•  Good  information  on  what  services  are  excluded  from 
PBSA 

•  There  is  still  some  confusion  in  the  field 

•  ALC  wants  repair  services  exempt  from  PBSA 

•  Product  Center:  “an  aversion  to  calling  the  process 
of  design,  development,  and  production  a  ‘service’” 

•  Product  Center:  “R&D  is  not  under  SCA” 

•  Product  Center:  logistics  personnel  don’t  have 
experience  with  SCA  and  fear  it 

•  Some  seem  to  think  PBSA  should  only  apply  to  services 
governed  by  the  SCA 
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To  summarize  the  information  in  this  appendix,  we  have  seen  that  there 
are  two  separate  questions  related  to  the  application  of  PBSA  to  the 
acquisition  of  services: 

•  What  constitutes  a  service? 

•  To  which  services  does  PBSA  apply? 

The  information  about  what  constitutes  a  service  is  reasonably  clear  in  the 
CFR  and  the  FARs,  although  FAR  Part  37  muddies  the  water  a  bit.  The 
exemptions  listed  in  FAR  Part  37,  AFI  63-124,  and  the  PBSA 
Implementation  Plan  are  also  clear,  though  the  reasons  for  differences  in 
the  lists  are  not.  Flowever,  our  discussions  at  the  ALC  and  the  Product 
Center  raised  some  points  about  these  two  issues  that  need  clarification. 

For  example,  we  heard  at  the  ALC  that  some  people  wanted  their  repair 
services  to  be  exempt  from  PBSA  “because  repair  services  are  under 
Walsh-Healey."  This  remark,  which  implies  the  opinion  that  ALC  repair 
services  constitute  "remanufacturing,"  gave  us  the  impression  that  the 
ALC  felt  the  application  of  PBSA  to  "services"  should  only  be  to  services 
that  are  covered  by  the  Service  Contract  Act.  At  the  Product  Center,  a 
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senior  executive  noted  that  there  was  an  "aversion"  to  calling  the  process 
of  design,  development,  and  production  of  specialized  products  a 
"service."  Since  we  had  heard  other  comments  about  the  distinction 
between  "professional"  and  "nonprofessional"  services,  this  remark  may 
also  have  been  an  indication  of  an  attitude  that  "services"  are  only  those 
areas  covered  by  the  SCA,  and  that  these  were  the  types  of  services  to 
which  PBS  A  should  be  applied.  The  last  two  comments  shown  in  the 
chart  relate  similar  attitudes. 

Perhaps  these  comments  reflect  an  understanding  at  the  ALC  and  the 
Product  Center  that  PBSA  is  generally  easier  to  implement  in  operational 
services  (e.g.,  grounds  maintenance).  These  are  services  for  which 
performance  thresholds  are  easily  defined  and  services  accomplished  by 
people  with  generally  lower  skill  levels  than  those  who  perform  other 
services  at  the  ALCs  and  Product  Centers.  These  lower-skilled  people  are 
exactly  those  whom  the  SCA  was  designed  to  protect. 
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